Compare commits

..

11 Commits

Author SHA1 Message Date
2873130259 Release 1.70.0 (#457) 2021-11-07 18:38:29 +01:00
d999a27159 Feature/move scraper configuration to symbol profile (#456)
* Move scraper configuration

* Update changelog
2021-11-07 18:36:28 +01:00
b6902e10ea Feature/improve import json file validation (#455)
* Improve import validation

* Update changelog
2021-11-07 17:16:19 +01:00
7f3f75386d Release 1.69.0 (#454) 2021-11-07 09:45:19 +01:00
678544748a Add symbol mapping (#452)
* Add symbol mapping

* Update changelog
2021-11-07 09:42:36 +01:00
632f3e3872 Add ok.csv (#453) 2021-11-06 21:09:06 +01:00
87301ddbd5 Feature/improve registration page (#451)
* Improve registration page

* Update changelog
2021-11-02 21:49:57 +01:00
7d03c373ac Release 1.68.0 (#450) 2021-11-01 21:27:23 +01:00
edb66bb166 Feature/extend statistics (#449)
* Extend statistics

* Update changelog
2021-11-01 21:15:09 +01:00
54bbc8446b Feature/prettify scraper symbol in chart (#448)
* Prettify scraper symbol in chart

* Update changelog
2021-11-01 20:29:16 +01:00
9933967e42 Release 1.67.0 (#447) 2021-11-01 19:33:45 +01:00
33 changed files with 323 additions and 161 deletions

View File

@ -5,7 +5,40 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## Unreleased ## 1.70.0 - 07.11.2021
### Changed
- Improved the validation of `json` files in the import functionality for transactions
- Moved the scraper configuration to the symbol profile model
### Todo
- Apply data migration (`yarn database:migrate`)
## 1.69.0 - 07.11.2021
### Added
- Added the symbol mapping attribute to the symbol profile model
### Changed
- Improved the registration page
### Todo
- Apply data migration (`yarn database:migrate`)
## 1.68.0 - 01.11.2021
### Changed
- Prettified the generic scraper symbols in the portfolio proportion chart component
- Extended the statistics section on the about page by the active users count (7d)
- Extended the statistics section on the about page by the new users count
## 1.67.0 - 31.10.2021
### Added ### Added
@ -52,7 +85,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Todo ### Todo
- Apply data migration (`yarn prisma migrate deploy`) - Apply data migration (`yarn database:migrate`)
## 1.62.0 - 17.10.2021 ## 1.62.0 - 17.10.2021
@ -150,7 +183,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Todo ### Todo
- Apply data migration (`yarn prisma migrate deploy`) - Apply data migration (`yarn database:migrate`)
## 1.55.0 - 20.09.2021 ## 1.55.0 - 20.09.2021
@ -165,7 +198,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Todo ### Todo
- Apply data migration (`yarn prisma migrate deploy`) - Apply data migration (`yarn database:migrate`)
## 1.54.0 - 18.09.2021 ## 1.54.0 - 18.09.2021
@ -186,7 +219,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Todo ### Todo
- Apply data migration (`yarn prisma migrate deploy`) - Apply data migration (`yarn database:migrate`)
## 1.53.0 - 13.09.2021 ## 1.53.0 - 13.09.2021
@ -308,7 +341,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Todo ### Todo
- Apply data migration (`yarn prisma migrate deploy`) - Apply data migration (`yarn database:migrate`)
## 1.41.0 - 21.08.2021 ## 1.41.0 - 21.08.2021
@ -361,7 +394,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Todo ### Todo
- Apply data migration (`yarn prisma migrate deploy`) - Apply data migration (`yarn database:migrate`)
## 1.38.0 - 14.08.2021 ## 1.38.0 - 14.08.2021
@ -421,7 +454,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Todo ### Todo
- Apply data migration (`yarn prisma migrate deploy`) - Apply data migration (`yarn database:migrate`)
## 1.34.0 - 07.08.2021 ## 1.34.0 - 07.08.2021

View File

@ -101,7 +101,7 @@ docker-compose -f docker/docker-compose-build-local.yml up
Run the following command to setup the database once Ghostfolio is running: Run the following command to setup the database once Ghostfolio is running:
```bash ```bash
docker-compose -f docker/docker-compose-build-local.yml exec ghostfolio yarn setup:database docker-compose -f docker/docker-compose-build-local.yml exec ghostfolio yarn database:setup
``` ```
### Fetch Historical Data ### Fetch Historical Data
@ -112,6 +112,14 @@ Open http://localhost:3333 in your browser and accomplish these steps:
1. Go to the _Admin Control Panel_ and click _Gather All Data_ to fetch historical data 1. Go to the _Admin Control Panel_ and click _Gather All Data_ to fetch historical data
1. Click _Sign out_ and check out the _Live Demo_ 1. Click _Sign out_ and check out the _Live Demo_
### Migrate Database
With the following command you can keep your database schema in sync after a Ghostfolio version update:
```bash
docker-compose -f docker/docker-compose-build-local.yml exec ghostfolio yarn database:migrate
```
## Development ## Development
### Prerequisites ### Prerequisites
@ -126,7 +134,7 @@ Open http://localhost:3333 in your browser and accomplish these steps:
1. Run `cd docker` 1. Run `cd docker`
1. Run `docker compose up -d` to start [PostgreSQL](https://www.postgresql.org) and [Redis](https://redis.io) 1. Run `docker compose up -d` to start [PostgreSQL](https://www.postgresql.org) and [Redis](https://redis.io)
1. Run `cd -` to go back to the project root directory 1. Run `cd -` to go back to the project root directory
1. Run `yarn setup:database` to initialize the database schema and populate your database with (example) data 1. Run `yarn database:setup` to initialize the database schema and populate your database with (example) data
1. Start server and client (see [_Development_](#Development)) 1. Start server and client (see [_Development_](#Development))
1. Login as _Admin_ with the following _Security Token_: `ae76872ae8f3419c6d6f64bf51888ecbcc703927a342d815fafe486acdb938da07d0cf44fca211a0be74a423238f535362d390a41e81e633a9ce668a6e31cdf9` 1. Login as _Admin_ with the following _Security Token_: `ae76872ae8f3419c6d6f64bf51888ecbcc703927a342d815fafe486acdb938da07d0cf44fca211a0be74a423238f535362d390a41e81e633a9ce668a6e31cdf9`
1. Go to the _Admin Control Panel_ and click _Gather All Data_ to fetch historical data 1. Go to the _Admin Control Panel_ and click _Gather All Data_ to fetch historical data

View File

@ -6,6 +6,7 @@ import { DataGatheringService } from '@ghostfolio/api/services/data-gathering.se
import { DataProviderModule } from '@ghostfolio/api/services/data-provider/data-provider.module'; import { DataProviderModule } from '@ghostfolio/api/services/data-provider/data-provider.module';
import { ExchangeRateDataModule } from '@ghostfolio/api/services/exchange-rate-data.module'; import { ExchangeRateDataModule } from '@ghostfolio/api/services/exchange-rate-data.module';
import { PrismaService } from '@ghostfolio/api/services/prisma.service'; import { PrismaService } from '@ghostfolio/api/services/prisma.service';
import { SymbolProfileModule } from '@ghostfolio/api/services/symbol-profile.module';
import { Module } from '@nestjs/common'; import { Module } from '@nestjs/common';
import { CacheController } from './cache.controller'; import { CacheController } from './cache.controller';
@ -15,7 +16,8 @@ import { CacheController } from './cache.controller';
DataGatheringModule, DataGatheringModule,
DataProviderModule, DataProviderModule,
ExchangeRateDataModule, ExchangeRateDataModule,
RedisCacheModule RedisCacheModule,
SymbolProfileModule
], ],
controllers: [CacheController], controllers: [CacheController],
providers: [ providers: [

View File

@ -4,6 +4,7 @@ import { DataGatheringService } from '@ghostfolio/api/services/data-gathering.se
import { DataProviderModule } from '@ghostfolio/api/services/data-provider/data-provider.module'; import { DataProviderModule } from '@ghostfolio/api/services/data-provider/data-provider.module';
import { ExchangeRateDataModule } from '@ghostfolio/api/services/exchange-rate-data.module'; import { ExchangeRateDataModule } from '@ghostfolio/api/services/exchange-rate-data.module';
import { PrismaService } from '@ghostfolio/api/services/prisma.service'; import { PrismaService } from '@ghostfolio/api/services/prisma.service';
import { SymbolProfileModule } from '@ghostfolio/api/services/symbol-profile.module';
import { Module } from '@nestjs/common'; import { Module } from '@nestjs/common';
import { JwtModule } from '@nestjs/jwt'; import { JwtModule } from '@nestjs/jwt';
@ -18,7 +19,8 @@ import { InfoService } from './info.service';
JwtModule.register({ JwtModule.register({
secret: process.env.JWT_SECRET_KEY, secret: process.env.JWT_SECRET_KEY,
signOptions: { expiresIn: '30 days' } signOptions: { expiresIn: '30 days' }
}) }),
SymbolProfileModule
], ],
controllers: [InfoController], controllers: [InfoController],
providers: [ providers: [

View File

@ -136,6 +136,28 @@ export class InfoService {
} }
} }
private async countNewUsers(aDays: number) {
return await this.prismaService.user.count({
orderBy: {
createdAt: 'desc'
},
where: {
AND: [
{
NOT: {
Analytics: null
}
},
{
createdAt: {
gt: subDays(new Date(), aDays)
}
}
]
}
});
}
private getDemoAuthToken() { private getDemoAuthToken() {
return this.jwtService.sign({ return this.jwtService.sign({
id: InfoService.DEMO_USER_ID id: InfoService.DEMO_USER_ID
@ -155,15 +177,19 @@ export class InfoService {
} }
const activeUsers1d = await this.countActiveUsers(1); const activeUsers1d = await this.countActiveUsers(1);
const activeUsers7d = await this.countActiveUsers(7);
const activeUsers30d = await this.countActiveUsers(30); const activeUsers30d = await this.countActiveUsers(30);
const newUsers30d = await this.countNewUsers(30);
const gitHubContributors = await this.countGitHubContributors(); const gitHubContributors = await this.countGitHubContributors();
const gitHubStargazers = await this.countGitHubStargazers(); const gitHubStargazers = await this.countGitHubStargazers();
return { return {
activeUsers1d, activeUsers1d,
activeUsers7d,
activeUsers30d, activeUsers30d,
gitHubContributors, gitHubContributors,
gitHubStargazers gitHubStargazers,
newUsers30d
}; };
} }

View File

@ -8,7 +8,7 @@ import { DataProviderModule } from '@ghostfolio/api/services/data-provider/data-
import { ExchangeRateDataModule } from '@ghostfolio/api/services/exchange-rate-data.module'; import { ExchangeRateDataModule } from '@ghostfolio/api/services/exchange-rate-data.module';
import { ImpersonationModule } from '@ghostfolio/api/services/impersonation.module'; import { ImpersonationModule } from '@ghostfolio/api/services/impersonation.module';
import { PrismaModule } from '@ghostfolio/api/services/prisma.module'; import { PrismaModule } from '@ghostfolio/api/services/prisma.module';
import { SymbolProfileService } from '@ghostfolio/api/services/symbol-profile.service'; import { SymbolProfileModule } from '@ghostfolio/api/services/symbol-profile.module';
import { Module } from '@nestjs/common'; import { Module } from '@nestjs/common';
import { CurrentRateService } from './current-rate.service'; import { CurrentRateService } from './current-rate.service';
@ -27,6 +27,7 @@ import { RulesService } from './rules.service';
ImpersonationModule, ImpersonationModule,
OrderModule, OrderModule,
PrismaModule, PrismaModule,
SymbolProfileModule,
UserModule UserModule
], ],
controllers: [PortfolioController], controllers: [PortfolioController],
@ -35,8 +36,7 @@ import { RulesService } from './rules.service';
CurrentRateService, CurrentRateService,
MarketDataService, MarketDataService,
PortfolioService, PortfolioService,
RulesService, RulesService
SymbolProfileService
] ]
}) })
export class PortfolioModule {} export class PortfolioModule {}

View File

@ -6,6 +6,7 @@ import { PrismaModule } from '@ghostfolio/api/services/prisma.module';
import { Module } from '@nestjs/common'; import { Module } from '@nestjs/common';
import { ExchangeRateDataModule } from './exchange-rate-data.module'; import { ExchangeRateDataModule } from './exchange-rate-data.module';
import { SymbolProfileModule } from './symbol-profile.module';
@Module({ @Module({
imports: [ imports: [
@ -13,7 +14,8 @@ import { ExchangeRateDataModule } from './exchange-rate-data.module';
DataEnhancerModule, DataEnhancerModule,
DataProviderModule, DataProviderModule,
ExchangeRateDataModule, ExchangeRateDataModule,
PrismaModule PrismaModule,
SymbolProfileModule
], ],
providers: [DataGatheringService], providers: [DataGatheringService],
exports: [DataEnhancerModule, DataGatheringService] exports: [DataEnhancerModule, DataGatheringService]

View File

@ -1,3 +1,4 @@
import { SymbolProfileService } from '@ghostfolio/api/services/symbol-profile.service';
import { import {
benchmarks, benchmarks,
ghostfolioFearAndGreedIndexSymbol ghostfolioFearAndGreedIndexSymbol
@ -32,7 +33,8 @@ export class DataGatheringService {
private readonly dataProviderService: DataProviderService, private readonly dataProviderService: DataProviderService,
private readonly exchangeRateDataService: ExchangeRateDataService, private readonly exchangeRateDataService: ExchangeRateDataService,
private readonly ghostfolioScraperApi: GhostfolioScraperApiService, private readonly ghostfolioScraperApi: GhostfolioScraperApiService,
private readonly prismaService: PrismaService private readonly prismaService: PrismaService,
private readonly symbolProfileService: SymbolProfileService
) {} ) {}
public async gather7Days() { public async gather7Days() {
@ -132,13 +134,22 @@ export class DataGatheringService {
} }
const currentData = await this.dataProviderService.get(dataGatheringItems); const currentData = await this.dataProviderService.get(dataGatheringItems);
const symbolProfiles = await this.symbolProfileService.getSymbolProfiles(
dataGatheringItems.map(({ symbol }) => {
return symbol;
})
);
for (const [symbol, response] of Object.entries(currentData)) { for (const [symbol, response] of Object.entries(currentData)) {
const symbolMapping = symbolProfiles.find((symbolProfile) => {
return symbolProfile.symbol === symbol;
})?.symbolMapping;
for (const dataEnhancer of this.dataEnhancers) { for (const dataEnhancer of this.dataEnhancers) {
try { try {
currentData[symbol] = await dataEnhancer.enhance({ currentData[symbol] = await dataEnhancer.enhance({
response, response,
symbol symbol: symbolMapping[dataEnhancer.getName()] ?? symbol
}); });
} catch (error) { } catch (error) {
console.error(`Failed to enhance data for symbol ${symbol}`, error); console.error(`Failed to enhance data for symbol ${symbol}`, error);
@ -261,21 +272,6 @@ export class DataGatheringService {
} }
} }
public async getCustomSymbolsToGather(
startDate?: Date
): Promise<IDataGatheringItem[]> {
const scraperConfigurations =
await this.ghostfolioScraperApi.getScraperConfigurations();
return scraperConfigurations.map((scraperConfiguration) => {
return {
dataSource: DataSource.GHOSTFOLIO,
date: startDate,
symbol: scraperConfiguration.symbol
};
});
}
public async getIsInProgress() { public async getIsInProgress() {
return await this.prismaService.property.findUnique({ return await this.prismaService.property.findUnique({
where: { key: 'LOCKED_DATA_GATHERING' } where: { key: 'LOCKED_DATA_GATHERING' }
@ -332,6 +328,7 @@ export class DataGatheringService {
orderBy: [{ symbol: 'asc' }], orderBy: [{ symbol: 'asc' }],
select: { select: {
dataSource: true, dataSource: true,
scraperConfiguration: true,
symbol: true symbol: true
} }
}) })
@ -352,12 +349,8 @@ export class DataGatheringService {
}; };
}); });
const customSymbolsToGather =
await this.ghostfolioScraperApi.getCustomSymbolsToGather(startDate);
return [ return [
...this.getBenchmarksToGather(startDate), ...this.getBenchmarksToGather(startDate),
...customSymbolsToGather,
...currencyPairsToGather, ...currencyPairsToGather,
...symbolProfilesToGather ...symbolProfilesToGather
]; ];
@ -371,9 +364,6 @@ export class DataGatheringService {
}) })
)?.date ?? new Date(); )?.date ?? new Date();
const customSymbolsToGather =
await this.ghostfolioScraperApi.getCustomSymbolsToGather(startDate);
const currencyPairsToGather = this.exchangeRateDataService const currencyPairsToGather = this.exchangeRateDataService
.getCurrencyPairs() .getCurrencyPairs()
.map(({ dataSource, symbol }) => { .map(({ dataSource, symbol }) => {
@ -394,20 +384,19 @@ export class DataGatheringService {
select: { date: true }, select: { date: true },
take: 1 take: 1
}, },
scraperConfiguration: true,
symbol: true symbol: true
} }
}) })
).map((item) => { ).map((symbolProfile) => {
return { return {
dataSource: item.dataSource, ...symbolProfile,
date: item.Order?.[0]?.date ?? startDate, date: symbolProfile.Order?.[0]?.date ?? startDate
symbol: item.symbol
}; };
}); });
return [ return [
...this.getBenchmarksToGather(startDate), ...this.getBenchmarksToGather(startDate),
...customSymbolsToGather,
...currencyPairsToGather, ...currencyPairsToGather,
...symbolProfilesToGather ...symbolProfilesToGather
]; ];

View File

@ -70,4 +70,8 @@ export class TrackinsightDataEnhancerService implements DataEnhancerInterface {
return Promise.resolve(response); return Promise.resolve(response);
} }
public getName() {
return 'TRACKINSIGHT';
}
} }

View File

@ -4,13 +4,19 @@ import { GhostfolioScraperApiService } from '@ghostfolio/api/services/data-provi
import { RakutenRapidApiService } from '@ghostfolio/api/services/data-provider/rakuten-rapid-api/rakuten-rapid-api.service'; import { RakutenRapidApiService } from '@ghostfolio/api/services/data-provider/rakuten-rapid-api/rakuten-rapid-api.service';
import { YahooFinanceService } from '@ghostfolio/api/services/data-provider/yahoo-finance/yahoo-finance.service'; import { YahooFinanceService } from '@ghostfolio/api/services/data-provider/yahoo-finance/yahoo-finance.service';
import { PrismaModule } from '@ghostfolio/api/services/prisma.module'; import { PrismaModule } from '@ghostfolio/api/services/prisma.module';
import { SymbolProfileModule } from '@ghostfolio/api/services/symbol-profile.module';
import { Module } from '@nestjs/common'; import { Module } from '@nestjs/common';
import { AlphaVantageService } from './alpha-vantage/alpha-vantage.service'; import { AlphaVantageService } from './alpha-vantage/alpha-vantage.service';
import { DataProviderService } from './data-provider.service'; import { DataProviderService } from './data-provider.service';
@Module({ @Module({
imports: [ConfigurationModule, CryptocurrencyModule, PrismaModule], imports: [
ConfigurationModule,
CryptocurrencyModule,
PrismaModule,
SymbolProfileModule
],
providers: [ providers: [
AlphaVantageService, AlphaVantageService,
DataProviderService, DataProviderService,

View File

@ -1,5 +1,6 @@
import { LookupItem } from '@ghostfolio/api/app/symbol/interfaces/lookup-item.interface'; import { LookupItem } from '@ghostfolio/api/app/symbol/interfaces/lookup-item.interface';
import { PrismaService } from '@ghostfolio/api/services/prisma.service'; import { PrismaService } from '@ghostfolio/api/services/prisma.service';
import { SymbolProfileService } from '@ghostfolio/api/services/symbol-profile.service';
import { import {
DATE_FORMAT, DATE_FORMAT,
getYesterday, getYesterday,
@ -13,19 +14,20 @@ import * as cheerio from 'cheerio';
import { format } from 'date-fns'; import { format } from 'date-fns';
import { import {
IDataGatheringItem,
IDataProviderHistoricalResponse, IDataProviderHistoricalResponse,
IDataProviderResponse, IDataProviderResponse,
MarketState MarketState
} from '../../interfaces/interfaces'; } from '../../interfaces/interfaces';
import { DataProviderInterface } from '../interfaces/data-provider.interface'; import { DataProviderInterface } from '../interfaces/data-provider.interface';
import { ScraperConfig } from './interfaces/scraper-config.interface';
@Injectable() @Injectable()
export class GhostfolioScraperApiService implements DataProviderInterface { export class GhostfolioScraperApiService implements DataProviderInterface {
private static NUMERIC_REGEXP = /[-]{0,1}[\d]*[.,]{0,1}[\d]+/g; private static NUMERIC_REGEXP = /[-]{0,1}[\d]*[.,]{0,1}[\d]+/g;
public constructor(private readonly prismaService: PrismaService) {} public constructor(
private readonly prismaService: PrismaService,
private readonly symbolProfileService: SymbolProfileService
) {}
public canHandle(symbol: string) { public canHandle(symbol: string) {
return isGhostfolioScraperApiSymbol(symbol); return isGhostfolioScraperApiSymbol(symbol);
@ -39,9 +41,10 @@ export class GhostfolioScraperApiService implements DataProviderInterface {
} }
try { try {
const symbol = aSymbols[0]; const [symbol] = aSymbols;
const [symbolProfile] = await this.symbolProfileService.getSymbolProfiles(
const scraperConfig = await this.getScraperConfigurationBySymbol(symbol); [symbol]
);
const { marketPrice } = await this.prismaService.marketData.findFirst({ const { marketPrice } = await this.prismaService.marketData.findFirst({
orderBy: { orderBy: {
@ -55,7 +58,7 @@ export class GhostfolioScraperApiService implements DataProviderInterface {
return { return {
[symbol]: { [symbol]: {
marketPrice, marketPrice,
currency: scraperConfig?.currency, currency: symbolProfile?.currency,
dataSource: DataSource.GHOSTFOLIO, dataSource: DataSource.GHOSTFOLIO,
marketState: MarketState.delayed marketState: MarketState.delayed
} }
@ -67,25 +70,6 @@ export class GhostfolioScraperApiService implements DataProviderInterface {
return {}; return {};
} }
public async getCustomSymbolsToGather(
startDate?: Date
): Promise<IDataGatheringItem[]> {
const ghostfolioSymbolProfiles =
await this.prismaService.symbolProfile.findMany({
where: {
dataSource: DataSource.GHOSTFOLIO
}
});
return ghostfolioSymbolProfiles.map(({ dataSource, symbol }) => {
return {
dataSource,
symbol,
date: startDate
};
});
}
public async getHistorical( public async getHistorical(
aSymbols: string[], aSymbols: string[],
aGranularity: Granularity = 'day', aGranularity: Granularity = 'day',
@ -99,11 +83,11 @@ export class GhostfolioScraperApiService implements DataProviderInterface {
} }
try { try {
const symbol = aSymbols[0]; const [symbol] = aSymbols;
const [symbolProfile] = await this.symbolProfileService.getSymbolProfiles(
const scraperConfiguration = await this.getScraperConfigurationBySymbol( [symbol]
symbol
); );
const scraperConfiguration = symbolProfile?.scraperConfiguration;
const get = bent(scraperConfiguration?.url, 'GET', 'string', 200, {}); const get = bent(scraperConfiguration?.url, 'GET', 'string', 200, {});
@ -128,22 +112,6 @@ export class GhostfolioScraperApiService implements DataProviderInterface {
return {}; return {};
} }
public async getScraperConfigurations(): Promise<ScraperConfig[]> {
try {
const { value: scraperConfigString } =
await this.prismaService.property.findFirst({
select: {
value: true
},
where: { key: 'SCRAPER_CONFIG' }
});
return JSON.parse(scraperConfigString);
} catch {}
return [];
}
public getName(): DataSource { public getName(): DataSource {
return DataSource.GHOSTFOLIO; return DataSource.GHOSTFOLIO;
} }
@ -162,11 +130,4 @@ export class GhostfolioScraperApiService implements DataProviderInterface {
return undefined; return undefined;
} }
} }
private async getScraperConfigurationBySymbol(aSymbol: string) {
const scraperConfigurations = await this.getScraperConfigurations();
return scraperConfigurations.find((scraperConfiguration) => {
return scraperConfiguration.symbol === aSymbol;
});
}
} }

View File

@ -1,6 +0,0 @@
export interface ScraperConfig {
currency: string;
selector: string;
symbol: string;
url: string;
}

View File

@ -0,0 +1,4 @@
export interface ScraperConfiguration {
selector: string;
url: string;
}

View File

@ -8,4 +8,6 @@ export interface DataEnhancerInterface {
response: IDataProviderResponse; response: IDataProviderResponse;
symbol: string; symbol: string;
}): Promise<IDataProviderResponse>; }): Promise<IDataProviderResponse>;
getName(): string;
} }

View File

@ -1,3 +1,4 @@
import { ScraperConfiguration } from '@ghostfolio/api/services/data-provider/ghostfolio-scraper-api/interfaces/scraper-configuration.interface';
import { Country } from '@ghostfolio/common/interfaces/country.interface'; import { Country } from '@ghostfolio/common/interfaces/country.interface';
import { Sector } from '@ghostfolio/common/interfaces/sector.interface'; import { Sector } from '@ghostfolio/common/interfaces/sector.interface';
import { AssetClass, AssetSubClass, DataSource } from '@prisma/client'; import { AssetClass, AssetSubClass, DataSource } from '@prisma/client';
@ -5,13 +6,15 @@ import { AssetClass, AssetSubClass, DataSource } from '@prisma/client';
export interface EnhancedSymbolProfile { export interface EnhancedSymbolProfile {
assetClass: AssetClass; assetClass: AssetClass;
assetSubClass: AssetSubClass; assetSubClass: AssetSubClass;
countries: Country[];
createdAt: Date; createdAt: Date;
currency: string | null; currency: string | null;
dataSource: DataSource; dataSource: DataSource;
id: string; id: string;
name: string | null; name: string | null;
updatedAt: Date; scraperConfiguration?: ScraperConfiguration;
symbol: string;
countries: Country[];
sectors: Sector[]; sectors: Sector[];
symbol: string;
symbolMapping?: { [key: string]: string };
updatedAt: Date;
} }

View File

@ -0,0 +1,11 @@
import { PrismaModule } from '@ghostfolio/api/services/prisma.module';
import { Module } from '@nestjs/common';
import { SymbolProfileService } from './symbol-profile.service';
@Module({
imports: [PrismaModule],
providers: [SymbolProfileService],
exports: [SymbolProfileService]
})
export class SymbolProfileModule {}

View File

@ -7,6 +7,8 @@ import { Injectable } from '@nestjs/common';
import { Prisma, SymbolProfile } from '@prisma/client'; import { Prisma, SymbolProfile } from '@prisma/client';
import { continents, countries } from 'countries-list'; import { continents, countries } from 'countries-list';
import { ScraperConfiguration } from './data-provider/ghostfolio-scraper-api/interfaces/scraper-configuration.interface';
@Injectable() @Injectable()
export class SymbolProfileService { export class SymbolProfileService {
constructor(private readonly prismaService: PrismaService) {} constructor(private readonly prismaService: PrismaService) {}
@ -29,7 +31,9 @@ export class SymbolProfileService {
return symbolProfiles.map((symbolProfile) => ({ return symbolProfiles.map((symbolProfile) => ({
...symbolProfile, ...symbolProfile,
countries: this.getCountries(symbolProfile), countries: this.getCountries(symbolProfile),
sectors: this.getSectors(symbolProfile) scraperConfiguration: this.getScraperConfiguration(symbolProfile),
sectors: this.getSectors(symbolProfile),
symbolMapping: this.getSymbolMapping(symbolProfile)
})); }));
} }
@ -49,6 +53,18 @@ export class SymbolProfileService {
); );
} }
private getScraperConfiguration(
symbolProfile: SymbolProfile
): ScraperConfiguration {
const scraperConfiguration =
symbolProfile.scraperConfiguration as Prisma.JsonObject;
return {
selector: scraperConfiguration.selector as string,
url: scraperConfiguration.url as string
};
}
private getSectors(symbolProfile: SymbolProfile): Sector[] { private getSectors(symbolProfile: SymbolProfile): Sector[] {
return ((symbolProfile?.sectors as Prisma.JsonArray) ?? []).map( return ((symbolProfile?.sectors as Prisma.JsonArray) ?? []).map(
(sector) => { (sector) => {
@ -61,4 +77,12 @@ export class SymbolProfileService {
} }
); );
} }
private getSymbolMapping(symbolProfile: SymbolProfile) {
return (
(symbolProfile['symbolMapping'] as {
[key: string]: string;
}) ?? {}
);
}
} }

View File

@ -229,7 +229,13 @@
mat-button mat-button
[routerLink]="['/']" [routerLink]="['/']"
> >
<gf-logo [hideName]="!currentRoute || currentRoute === 'start'"></gf-logo> <gf-logo
[hideName]="
!currentRoute ||
currentRoute === 'register' ||
currentRoute === 'start'
"
></gf-logo>
</a> </a>
<span class="spacer"></span> <span class="spacer"></span>
<a <a

View File

@ -107,7 +107,7 @@
<mat-card> <mat-card>
<mat-card-content> <mat-card-content>
<div class="row"> <div class="row">
<div class="col-xs-12 col-md-3 my-2"> <div class="col-xs-12 col-md-4 my-2">
<h3 class="mb-0" [hidden]="!statistics?.activeUsers1d"> <h3 class="mb-0" [hidden]="!statistics?.activeUsers1d">
{{ statistics?.activeUsers1d ?? '-' }} {{ statistics?.activeUsers1d ?? '-' }}
</h3> </h3>
@ -117,7 +117,17 @@
> >
</div> </div>
</div> </div>
<div class="col-xs-12 col-md-3 my-2"> <div class="col-xs-12 col-md-4 my-2">
<h3 class="mb-0" [hidden]="!statistics?.activeUsers7d">
{{ statistics?.activeUsers7d ?? '-' }}
</h3>
<div class="h6 mb-0">
<span i18n>Active Users</span>&nbsp;<small class="text-muted"
>(Last 7 days)</small
>
</div>
</div>
<div class="col-xs-12 col-md-4 my-2">
<h3 class="mb-0" [hidden]="!statistics?.activeUsers30d"> <h3 class="mb-0" [hidden]="!statistics?.activeUsers30d">
{{ statistics?.activeUsers30d ?? '-' }} {{ statistics?.activeUsers30d ?? '-' }}
</h3> </h3>
@ -127,13 +137,23 @@
> >
</div> </div>
</div> </div>
<div class="col-xs-12 col-md-3 my-2"> <div class="col-xs-12 col-md-4 my-2">
<h3 class="mb-0" [hidden]="!statistics?.newUsers30d">
{{ statistics?.newUsers30d ?? '-' }}
</h3>
<div class="h6 mb-0">
<span i18n>New Users</span>&nbsp;<small class="text-muted"
>(Last 30 days)</small
>
</div>
</div>
<div class="col-xs-12 col-md-4 my-2">
<h3 class="mb-0" [hidden]="!statistics?.gitHubContributors"> <h3 class="mb-0" [hidden]="!statistics?.gitHubContributors">
{{ statistics?.gitHubContributors ?? '-' }} {{ statistics?.gitHubContributors ?? '-' }}
</h3> </h3>
<div class="h6 mb-0" i18n>Contributors on GitHub</div> <div class="h6 mb-0" i18n>Contributors on GitHub</div>
</div> </div>
<div class="col-xs-12 col-md-3 my-2"> <div class="col-xs-12 col-md-4 my-2">
<h3 class="mb-0" [hidden]="!statistics?.gitHubStargazers"> <h3 class="mb-0" [hidden]="!statistics?.gitHubStargazers">
{{ statistics?.gitHubStargazers ?? '-' }} {{ statistics?.gitHubStargazers ?? '-' }}
</h3> </h3>

View File

@ -4,6 +4,7 @@ import { DataService } from '@ghostfolio/client/services/data.service';
import { ImpersonationStorageService } from '@ghostfolio/client/services/impersonation-storage.service'; import { ImpersonationStorageService } from '@ghostfolio/client/services/impersonation-storage.service';
import { UserService } from '@ghostfolio/client/services/user/user.service'; import { UserService } from '@ghostfolio/client/services/user/user.service';
import { UNKNOWN_KEY } from '@ghostfolio/common/config'; import { UNKNOWN_KEY } from '@ghostfolio/common/config';
import { prettifySymbol } from '@ghostfolio/common/helper';
import { import {
PortfolioDetails, PortfolioDetails,
PortfolioPosition, PortfolioPosition,
@ -246,9 +247,9 @@ export class AllocationsPageComponent implements OnDestroy, OnInit {
} }
if (position.assetClass === AssetClass.EQUITY) { if (position.assetClass === AssetClass.EQUITY) {
this.symbols[symbol] = { this.symbols[prettifySymbol(symbol)] = {
symbol,
name: position.name, name: position.name,
symbol: prettifySymbol(symbol),
value: aPeriod === 'original' ? position.investment : position.value value: aPeriod === 'original' ? position.investment : position.value
}; };
} }

View File

@ -12,6 +12,7 @@ import { User } from '@ghostfolio/common/interfaces';
import { hasPermission, permissions } from '@ghostfolio/common/permissions'; import { hasPermission, permissions } from '@ghostfolio/common/permissions';
import { DataSource, Order as OrderModel } from '@prisma/client'; import { DataSource, Order as OrderModel } from '@prisma/client';
import { format, parseISO } from 'date-fns'; import { format, parseISO } from 'date-fns';
import { isArray } from 'lodash';
import { DeviceDetectorService } from 'ngx-device-detector'; import { DeviceDetectorService } from 'ngx-device-detector';
import { Subject, Subscription } from 'rxjs'; import { Subject, Subscription } from 'rxjs';
import { takeUntil } from 'rxjs/operators'; import { takeUntil } from 'rxjs/operators';
@ -189,6 +190,11 @@ export class TransactionsPageComponent implements OnDestroy, OnInit {
try { try {
if (file.type === 'application/json') { if (file.type === 'application/json') {
const content = JSON.parse(fileContent); const content = JSON.parse(fileContent);
if (!isArray(content.orders)) {
throw new Error();
}
try { try {
await this.importTransactionsService.importJson({ await this.importTransactionsService.importJson({
content: content.orders, content: content.orders,

View File

@ -2,6 +2,7 @@ import { ChangeDetectorRef, Component, OnInit } from '@angular/core';
import { ActivatedRoute, Router } from '@angular/router'; import { ActivatedRoute, Router } from '@angular/router';
import { DataService } from '@ghostfolio/client/services/data.service'; import { DataService } from '@ghostfolio/client/services/data.service';
import { UNKNOWN_KEY } from '@ghostfolio/common/config'; import { UNKNOWN_KEY } from '@ghostfolio/common/config';
import { prettifySymbol } from '@ghostfolio/common/helper';
import { import {
PortfolioPosition, PortfolioPosition,
PortfolioPublicDetails PortfolioPublicDetails
@ -169,9 +170,9 @@ export class PublicPageComponent implements OnInit {
this.portfolioPublicDetails.holdings[symbol].value; this.portfolioPublicDetails.holdings[symbol].value;
} }
this.symbols[symbol] = { this.symbols[prettifySymbol(symbol)] = {
symbol,
name: position.name, name: position.name,
symbol: prettifySymbol(symbol),
value: position.value value: position.value
}; };
} }

View File

@ -1,4 +1,4 @@
import { ChangeDetectorRef, Component, OnDestroy, OnInit } from '@angular/core'; import { Component, OnDestroy, OnInit } from '@angular/core';
import { MatDialog } from '@angular/material/dialog'; import { MatDialog } from '@angular/material/dialog';
import { Router } from '@angular/router'; import { Router } from '@angular/router';
import { DataService } from '@ghostfolio/client/services/data.service'; import { DataService } from '@ghostfolio/client/services/data.service';
@ -6,6 +6,7 @@ import { TokenStorageService } from '@ghostfolio/client/services/token-storage.s
import { hasPermission, permissions } from '@ghostfolio/common/permissions'; import { hasPermission, permissions } from '@ghostfolio/common/permissions';
import { LineChartItem } from '@ghostfolio/ui/line-chart/interfaces/line-chart.interface'; import { LineChartItem } from '@ghostfolio/ui/line-chart/interfaces/line-chart.interface';
import { format } from 'date-fns'; import { format } from 'date-fns';
import { DeviceDetectorService } from 'ngx-device-detector';
import { Subject } from 'rxjs'; import { Subject } from 'rxjs';
import { takeUntil } from 'rxjs/operators'; import { takeUntil } from 'rxjs/operators';
@ -20,6 +21,7 @@ import { ShowAccessTokenDialog } from './show-access-token-dialog/show-access-to
export class RegisterPageComponent implements OnDestroy, OnInit { export class RegisterPageComponent implements OnDestroy, OnInit {
public currentYear = format(new Date(), 'yyyy'); public currentYear = format(new Date(), 'yyyy');
public demoAuthToken: string; public demoAuthToken: string;
public deviceType: string;
public hasPermissionForSocialLogin: boolean; public hasPermissionForSocialLogin: boolean;
public historicalDataItems: LineChartItem[]; public historicalDataItems: LineChartItem[];
@ -29,8 +31,8 @@ export class RegisterPageComponent implements OnDestroy, OnInit {
* @constructor * @constructor
*/ */
public constructor( public constructor(
private changeDetectorRef: ChangeDetectorRef,
private dataService: DataService, private dataService: DataService,
private deviceService: DeviceDetectorService,
private dialog: MatDialog, private dialog: MatDialog,
private router: Router, private router: Router,
private tokenStorageService: TokenStorageService private tokenStorageService: TokenStorageService
@ -45,6 +47,7 @@ export class RegisterPageComponent implements OnDestroy, OnInit {
const { demoAuthToken, globalPermissions } = this.dataService.fetchInfo(); const { demoAuthToken, globalPermissions } = this.dataService.fetchInfo();
this.demoAuthToken = demoAuthToken; this.demoAuthToken = demoAuthToken;
this.deviceType = this.deviceService.getDeviceInfo().deviceType;
this.hasPermissionForSocialLogin = hasPermission( this.hasPermissionForSocialLogin = hasPermission(
globalPermissions, globalPermissions,
permissions.enableSocialLogin permissions.enableSocialLogin

View File

@ -1,30 +1,46 @@
<div class="intro-container mb-5">
<div class="intro-inner-container mx-auto">
<div class="h-100 intro w-100"></div>
</div>
</div>
<div class="container"> <div class="container">
<div class="row"> <div class="row">
<div class="col"> <div
<h3 class="d-flex justify-content-center mb-3 text-center" i18n> class="align-items-center d-flex flex-column justify-content-center w-100"
Create your Account >
</h3> <gf-logo size="large"></gf-logo>
<mat-card class="mb-4"> <p class="lead m-0">Wealth Management Software</p>
<mat-card-content class="text-center"> </div>
<button </div>
class="d-inline-block"
color="primary" <div class="button-container row">
<div class="align-items-center col d-flex justify-content-center">
<div class="py-5 text-center">
<button
class="d-inline-block"
color="primary"
i18n
mat-flat-button
[disabled]="!demoAuthToken"
(click)="createAccount()"
>
Create Account
</button>
<ng-container *ngIf="hasPermissionForSocialLogin">
<div
class="m-3 text-muted"
i18n i18n
mat-flat-button [ngClass]="{'d-inline': deviceType !== 'mobile' }"
[disabled]="!demoAuthToken"
(click)="createAccount()"
> >
Create Account or
</button> </div>
<ng-container *ngIf="hasPermissionForSocialLogin"> <a color="accent" href="/api/auth/google" mat-flat-button
<div class="my-3 text-muted" i18n>or</div> ><ion-icon class="mr-1" name="logo-google"></ion-icon
<a color="accent" href="/api/auth/google" mat-flat-button ><span i18n>Continue with Google</span></a
><ion-icon class="mr-1" name="logo-google"></ion-icon >
><span i18n>Continue with Google</span></a </ng-container>
> </div>
</ng-container>
</mat-card-content>
</mat-card>
</div> </div>
</div> </div>
</div> </div>

View File

@ -1,3 +1,26 @@
:host { :host {
display: block; display: block;
.button-container {
.mat-stroked-button {
background-color: var(--light-background);
}
}
.intro-container {
background-color: #ffffff;
margin-top: -5rem;
.intro-inner-container {
aspect-ratio: 16 / 9;
max-height: 66vh;
.intro {
background-image: url('/assets/intro.jpg');
background-position: top left;
background-repeat: no-repeat;
background-size: contain;
}
}
}
} }

View File

@ -1,11 +1,11 @@
import { Pipe, PipeTransform } from '@angular/core'; import { Pipe, PipeTransform } from '@angular/core';
import { ghostfolioScraperApiSymbolPrefix } from '@ghostfolio/common/config'; import { prettifySymbol } from '@ghostfolio/common/helper';
@Pipe({ name: 'gfSymbol' }) @Pipe({ name: 'gfSymbol' })
export class SymbolPipe implements PipeTransform { export class SymbolPipe implements PipeTransform {
public constructor() {} public constructor() {}
public transform(aSymbol: string): string { public transform(aSymbol: string) {
return aSymbol?.replace(ghostfolioScraperApiSymbolPrefix, ''); return prettifySymbol(aSymbol);
} }
} }

View File

@ -116,3 +116,7 @@ export const DATE_FORMAT = 'yyyy-MM-dd';
export function parseDate(date: string) { export function parseDate(date: string) {
return parse(date, DATE_FORMAT, new Date()); return parse(date, DATE_FORMAT, new Date());
} }
export function prettifySymbol(aSymbol: string): string {
return aSymbol?.replace(ghostfolioScraperApiSymbolPrefix, '');
}

View File

@ -1,6 +1,8 @@
export interface Statistics { export interface Statistics {
activeUsers1d: number; activeUsers1d: number;
activeUsers7d: number;
activeUsers30d: number; activeUsers30d: number;
gitHubContributors: number; gitHubContributors: number;
gitHubStargazers: number; gitHubStargazers: number;
newUsers30d: number;
} }

View File

@ -1,6 +1,6 @@
{ {
"name": "ghostfolio", "name": "ghostfolio",
"version": "1.66.0", "version": "1.70.0",
"homepage": "https://ghostfol.io", "homepage": "https://ghostfol.io",
"license": "AGPL-3.0", "license": "AGPL-3.0",
"scripts": { "scripts": {
@ -19,8 +19,10 @@
"database:format-schema": "prisma format", "database:format-schema": "prisma format",
"database:generate-typings": "prisma generate", "database:generate-typings": "prisma generate",
"database:gui": "prisma studio", "database:gui": "prisma studio",
"database:migrate": "prisma migrate deploy",
"database:push": "prisma db push", "database:push": "prisma db push",
"database:seed": "prisma db seed --preview-feature", "database:seed": "prisma db seed --preview-feature",
"database:setup": "yarn database:push && yarn database:seed",
"dep-graph": "nx dep-graph", "dep-graph": "nx dep-graph",
"e2e": "ng e2e", "e2e": "ng e2e",
"format": "nx format:write", "format": "nx format:write",
@ -33,7 +35,6 @@
"nx": "nx", "nx": "nx",
"postinstall": "prisma generate && ngcc --properties es2015 browser module main", "postinstall": "prisma generate && ngcc --properties es2015 browser module main",
"replace-placeholders-in-build": "node ./replace.build.js", "replace-placeholders-in-build": "node ./replace.build.js",
"setup:database": "yarn database:push && yarn database:seed",
"start": "node dist/apps/api/main", "start": "node dist/apps/api/main",
"start:client": "ng serve client --hmr -o", "start:client": "ng serve client --hmr -o",
"start:prod": "node apps/api/main", "start:prod": "node apps/api/main",

View File

@ -0,0 +1,2 @@
-- AlterTable
ALTER TABLE "SymbolProfile" ADD COLUMN "symbolMapping" JSONB;

View File

@ -0,0 +1,2 @@
-- AlterTable
ALTER TABLE "SymbolProfile" ADD COLUMN "scraperConfiguration" JSONB;

View File

@ -118,18 +118,20 @@ model Settings {
} }
model SymbolProfile { model SymbolProfile {
assetClass AssetClass? assetClass AssetClass?
assetSubClass AssetSubClass? assetSubClass AssetSubClass?
countries Json? countries Json?
createdAt DateTime @default(now()) createdAt DateTime @default(now())
currency String? currency String?
dataSource DataSource dataSource DataSource
id String @id @default(uuid()) id String @id @default(uuid())
name String? name String?
Order Order[] Order Order[]
updatedAt DateTime @updatedAt updatedAt DateTime @updatedAt
sectors Json? scraperConfiguration Json?
symbol String sectors Json?
symbol String
symbolMapping Json?
@@unique([dataSource, symbol]) @@unique([dataSource, symbol])
} }

2
test/import/ok.csv Normal file
View File

@ -0,0 +1,2 @@
Date,Code,Currency,Price,Quantity,Action,Fee
16/09/2021,MSFT,USD,298.580,5,buy,19.00
1 Date Code Currency Price Quantity Action Fee
2 16/09/2021 MSFT USD 298.580 5 buy 19.00