rakam-api

That's correct but there is also an alternative approach. You can track your user event data into your database using an open-source solution such as Rakam API (https://github.com/rakam-io/rakam) or Snowplow (https://github.com/snowplow/snowplow) and use a product analytics tool such as Rakam (https://rakam.io/product) that provides a set of prebuilt reports & ad-hoc reporting interface for the data on top of your database. That way, you won't be sharing your data with a third-party company, have full control over your data, be able to access an interface similar to Google Analytics with prebuilt reports and also have the ability to model your data for your custom needs and do custom reporting. Shameless plug: I'm the maintainer of the Rakam API and working for the company behind that open-source product.

We also developed a product that collects the user event data via SDKs (https://github.com/rakam-io/rakam), depending on the data volume the users pick one of the two deployment types. The first one is based on Postgresql 11 and it takes advantage of partitioned tables (one table for each event time for the configured time period such as month, week, etc.), BRIN indexes, and materialized views. It's capable of handling 250M events per month without an issue but beyond that we suggest the users to use data-warehouse that supports horizontal scaling and columnar storage engine so we use Snowflake with Snowpipe, Kinesis or Kafka (pluggable) and S3. I have also tried using Clickhouse which is much cheaper but it lacks tooling for automatic scaling.

That's what we do at Rakam. If you have less than ~10M events per month, Postgresql works smoothly if you use features such as partitioned tables, parallel queries, and BRIN indexes. The only limitation is that since it's not horizontally scalable, your data must fit in one server. We have SDKs that provide ways to send the event data with the following format: rakam.logEvent('pageview', {url: 'https://test.com'}) The event types and attributes are all dynamic. The API server automatically infers the attribute types from the JSON blob and creates the tables which correspond to event types and the columns which correspond to event attributes and inserts the data into that table. It also enriches the events with visitor information such as user agent, location, referrer, etc. The users just run the following SQL query: SELECT url, count(*)

That's actually what we do at Rakam. Postgresql fits in many analytics workloads with partitioned tables, parallel queries, and BRIN indexes. The only limitation is that since it's not horizontally scalable, your data must fit in one server. `it just works` up to ~10M events per month. The SDKs provide ways to send the event data with the following format: rakam.logEvent('pageview', {url: 'https://test.com'}) The event types and attributes are all dynamic. The API server automatically infers the attribute types from the JSON blob and creates the tables which correspond to event types and the columns which correspond to event attributes and inserts the data into that table. It also enriches the events with visitor information such as user agent, location, referrer, etc. The users just run the following SQL query: SELECT url, count(*) from

I have an open-source project that collects the customer events via SDKs and stores it in a data warehouse. It's a distributed system, the mobile and web SDKs batch the user events on their devices and push it to our API in JSON format. The API servers enrich & sanitize the data, validate the schema, convert it to a serialized AVRO binary, and push it to a commit-log system such as Kinesis or Kafka (It's pluggable). We have another project that fetches data from Kafka & Kinesis in small batches, converts the data into columnar format and stores it in an S3 bucket / Google Cloud Storage. Then, we integrate their preferred data-warehouse into their distributed filesystem. That way they have all their raw data in their infrastructure for other systems such as fraud detection, recommendation, etc. but they have SQL access to their data as wel

Hey there, Could you please explain your use-case? Depending on your use-case, there are multiple solutions that fit in different use-cases. Plug & Play Solutions: Matomo: It's basically on-premise Google Analytics, it stores the data on Mysql and provides pageview & event metrics. (https://github.com/matomo-org/matomo) Countly: It's the extensible version of Matomo on top of Mongodb. While storing the raw data might not be their best bet, if you're looking for event analytics dashboard with pre-built features, it's definitely worth trying out. (https://github.com/Countly/countly-server) Self-service Solutions: Snowplow: It's an enterprise level event analytics solution. It's really extensible and it supports many different data-warehouse solutions such as Elasticsearch, Redshift, etc. but not that easy to learn & maintain. (https://githu

The long term solution is usually building an in-house analytics service but it's not easy and cheap. We're also developing a generic analytics platform that can be installed with one-click to your favorite cloud provider so that you can store your event data in your database and build an internal analytics service without coding. Check out: https://github.com/rakam-io/rakam

Data management is a hard problem because most of the companies have different requirements and culture, that's why many "data-driven" companies try to build their in-house own analytics solution. In order to address this problem, we're working on an open-source data analytics platform that let's you to collect data from anywhere and manage it easily with our cloud. Unlike many other analytics services, it solves ETL, data warehouse and BI problem transparently to you so that you can customize them for your needs. It stores your data in optimized format with a schema and allows you to query your data with both SQL and complex behavioral analytics queries via RESTFul API. Check it out: https://github.com/rakam-io/rakam

Shameless plug: I'm the maintainer of Rakam project. You don't have to deal with all of these complexities. There are open-source projects that can setup that infrastructure for you. For example, we provide Cloudformation scripts that setup data analytics cluster for you, Kinesis, S3, PrestoDB and RESTFul API for collecting and querying data-sets so that you can use it similar to how you use a SaaS product. We also have an integrated visualization project that can connect your Rakam API and allow you to create dynamic reports and custom dashboards. You can run queries via user interface similar to Scube UI and also complex behavioral queries such as funnel and cohort queries as well as SQL. https://github.com/rakam-io/rakam

We're implementing Clickhouse (http://clickhouse.yandex) to our open-source analytics platform Rakam. (https://github.com/rakam-io/rakam)

Shameless plug: We're also working on an on-premise custom analytics platform that can be deployed either Heroku or AWS with Cloudformation. https://github.com/rakam-io/rakam