ApsaraCache: A Redis branch originated from Alibaba Group (opens in new tab)

(github.com)

43 pointslaithshadeed8y ago41 comments

41 comments

Hi all, I'm here in Hangzhou and had the pleasure to have a meeting with the team working at that. It's a great team and we'll work into merging stuff into upstream Redis when it makes sense. It's cool that companies other than Redis Labs are starting to contribute to the development!

Note that this branch is based on Redis 2.8 so of course porting stuff requires some work.

tyingq8y ago

>Note that this branch is based on Redis 2.8

Hmm. The Github repo says it's based on 4.0: "ApsaraCache is based on the Redis official release 4.0" (from https://github.com/alibaba/ApsaraCache/blob/develop/README.m...)

antirez8y ago

Oh probably I mistaken other patches I discussed that will need to be ported. Thanks for the reply.

ddorian438y ago

Can I ask what you think of pedis ? (see my other comment on the thread)

antirez8y ago

Hey, I never tested it, I've no idea. In general I think that threading Redis is a good idea and I'm going to add more threading very soon. I'm less sold on making disk the Redis primary storage...

gregoriol8y ago

Reminds me of HHVM: a specific solution developed by a specific team for their specific problem.

Might help Redis on the long run though: some ideas might be brought back to Redis, or at least it will bring some challenge to the devs.

Meanwhile, keep calm and use Redis ;-)

ianlevesque8y ago

Agreed, though HHVM went on to be deployed at some other small websites [1]. Maybe this will too.

1. http://hhvm.com/blog/7205/wikipedia-on-hhvm

mintplant8y ago

On the other hand, Wikipedia is going to have to migrate back off of HHVM now that Facebook is planning to break compatibility with PHP and focus totally on Hack [0]. So hopping on board someone else's solution can bite you if their priorities change.

[0] http://hhvm.com/blog/2017/09/18/the-future-of-hhvm.html

1 more reply

danjf8y ago

30% faster by which measurements? Not much info on the docs....

0T-T08y ago

yeah, one of the few red flags

> In short connection scenario, ApsaraCache makes 30% performance increase compared with the vanilla version.

sounds like a cherry picked case to me

detaro8y ago

Cherry-picked IMHO isn't necessarily the right label for that. They don't claim they made an universally 30% faster redis, they clearly say one of the things they did improved performance for a specific case.

If anything, the OP here is misleading by quoting it as a general improvement.

ehllo8y ago

for them it is a real need.

(Talk with Daniel Zang CEO Alibaba Group) https://youtu.be/YhlvtcBwxPM?t=326

olavgg8y ago

I just tested it with redis-benchmark. The performance is the same as with Redis.

tyingq8y ago

It mentions "in short connection scenario".

I take that to mean repeated connect/set-get/disconnect. Like you might get from a client in a fastcgi scenario...php for example. Maybe add the -k 0 flag to the benchmark and a low-ish number of concurrent clients.

danjf8y ago

Oh...

daotoad8y ago

This looks pretty interesting to me. But it leads to a lot of questions that are unanswered on the webpage.

The claims of 30% faster for a particular workload are reasonable, but what configuration is needed and what exact workload are we talking about? I'd love to see some benchmarking code.

What are the limitations imposed by running in memcached mode? Can you still do clustering?

What other changes are they planning? Site just says "many". Are we talking about memory allocation or persistence changes? Maybe additional interface compatibility modes? It's hard to bet on this when we don't have any idea what else it will bring to the table.

Where possible, I'd really like to see these improvements factored into distinct redis modules and made available as plugins to the core.

stonewhite8y ago

Goals: For both support Redis and Memcached protocol with no client code need to be modified. For highly stable and efficient in production environment.

At face value it appears to be a NIH driven development. It is not only a redis fork it is also a persistent memcached port, which is _only_ selectable on launch. Maybe there is a catch, but lack of a roadmap and/or design document certainly doesn't really help.

ksec8y ago

I remember some time ago Alibaba was the largest Memcached user on the planet, may be they wanted to smooth the operation and use Redis whenever possible?

pmontra8y ago

Why forking instead of submitting pull requests to Redis?

lugg8y ago

The readme states they have a few extra unreleased features.

I'm guessing their changes are for internal use and may be out of scope for redis or simply they want to share their internal fork and get feedback/help.

I know you didn't say anything to the contrary but I feel it needs to be said: forks aren't inherently bad, they're a sign of a mature and healthy open source ecosystem.

And I'm sure if it's worth it to someone, perf improvements will eventually get upstreamed.

helb8y ago

It seems they made a lot of changes, and some would be probably unsuitable for Redis. "Memcached mode" for example.

arthursilva8y ago

I wonder if Redis throughput is even a problem in practice. People usually add nodes to get more capacity (memory).

ihsw28y ago

For most use-cases, Redis nodes will saturate network IO before CPU, but it should be noted that Redis is single-threaded so YMMV.

joneholland8y ago

We do heavy pipelining and tend to max out the single thread, so we run multiple redis instances per host.

mangatmodi8y ago

One of the most useless FAQ I saw. I would have been more interested in knowing the improvements or changes made when compared to Redis. I had some trouble with large scale master slave Redis configuration. I was hoping if they have solved the issues.

detaro8y ago

They list the two things they changed compared to redis in the readme.

mangatmodi8y ago

I went back and read again. "ApsaraCache is based on the Redis official release 4.0 and has many features and performance enhancements" - I could find only memcache and performance improvement. Now Membache has nothing to do with Redis, its something not improving Redis, and 30% performance improvement is very vague statement with no other supporting benchmarks and documents. When they say many, I can't see any other than these two.

1 more reply

ddorian438y ago

I think they are also working on: https://github.com/fastio/pedis (redis implemented on top of seastar-framework / scylladb )

tuyguntn8y ago

any experience using it in production? looks promising results

ddorian438y ago

still missing many features so I don't think it's ready, you should ask on their issue queue

tyingq8y ago

Diffs of what they changed in /src from Redis 4.0: https://github.com/asaradiffs/apsaradiffs

tuananh8y ago

documentation and readme of alibaba projects are scary. Have anyone use this or any alibaba open source projects for production?

CyberDildonics8y ago

Does it still run on only one thread?

rvnx8y ago

This project is some kind of a joke;

Look at the commit history; https://github.com/alibaba/ApsaraCache/commits/develop

15 are "writing of README / adding my copyright", and 2 are actual feature (and not long)

zokier8y ago

I think this is what you are missing: https://github.com/alibaba/ApsaraCache/commit/07518063b17b13...

detaro8y ago

Kind of sad how things can't be partial and in the process of being open-sourced, no they have to be complete or they are "jokes".

dang8y ago

Please don't post uncharitable dismissals of other people's work to HN. If you read https://news.ycombinator.com/newsguidelines.html properly you'll see that doing this breaks more than one of the site guidelines.

j / k navigate · click thread line to collapse

41 comments

antirez8y ago

Note that this branch is based on Redis 2.8 so of course porting stuff requires some work.

tyingq8y ago

>Note that this branch is based on Redis 2.8

Hmm. The Github repo says it's based on 4.0: "ApsaraCache is based on the Redis official release 4.0" (from https://github.com/alibaba/ApsaraCache/blob/develop/README.m...)

antirez8y ago

Oh probably I mistaken other patches I discussed that will need to be ported. Thanks for the reply.

ddorian438y ago

Can I ask what you think of pedis ? (see my other comment on the thread)

antirez8y ago

Hey, I never tested it, I've no idea. In general I think that threading Redis is a good idea and I'm going to add more threading very soon. I'm less sold on making disk the Redis primary storage...

gregoriol8y ago

Reminds me of HHVM: a specific solution developed by a specific team for their specific problem.

Might help Redis on the long run though: some ideas might be brought back to Redis, or at least it will bring some challenge to the devs.

Meanwhile, keep calm and use Redis ;-)

ianlevesque8y ago

Agreed, though HHVM went on to be deployed at some other small websites [1]. Maybe this will too.

1. http://hhvm.com/blog/7205/wikipedia-on-hhvm

mintplant8y ago

[0] http://hhvm.com/blog/2017/09/18/the-future-of-hhvm.html

1 more reply

danjf8y ago

30% faster by which measurements? Not much info on the docs....

0T-T08y ago

yeah, one of the few red flags

> In short connection scenario, ApsaraCache makes 30% performance increase compared with the vanilla version.

sounds like a cherry picked case to me

detaro8y ago

If anything, the OP here is misleading by quoting it as a general improvement.

ehllo8y ago

for them it is a real need.

(Talk with Daniel Zang CEO Alibaba Group) https://youtu.be/YhlvtcBwxPM?t=326

olavgg8y ago

I just tested it with redis-benchmark. The performance is the same as with Redis.

tyingq8y ago

It mentions "in short connection scenario".

danjf8y ago

Oh...

daotoad8y ago

This looks pretty interesting to me. But it leads to a lot of questions that are unanswered on the webpage.

The claims of 30% faster for a particular workload are reasonable, but what configuration is needed and what exact workload are we talking about? I'd love to see some benchmarking code.

What are the limitations imposed by running in memcached mode? Can you still do clustering?

Where possible, I'd really like to see these improvements factored into distinct redis modules and made available as plugins to the core.

stonewhite8y ago

Goals: For both support Redis and Memcached protocol with no client code need to be modified. For highly stable and efficient in production environment.

ksec8y ago

I remember some time ago Alibaba was the largest Memcached user on the planet, may be they wanted to smooth the operation and use Redis whenever possible?

pmontra8y ago

Why forking instead of submitting pull requests to Redis?

lugg8y ago

The readme states they have a few extra unreleased features.

I'm guessing their changes are for internal use and may be out of scope for redis or simply they want to share their internal fork and get feedback/help.

I know you didn't say anything to the contrary but I feel it needs to be said: forks aren't inherently bad, they're a sign of a mature and healthy open source ecosystem.

And I'm sure if it's worth it to someone, perf improvements will eventually get upstreamed.

helb8y ago

It seems they made a lot of changes, and some would be probably unsuitable for Redis. "Memcached mode" for example.

arthursilva8y ago

I wonder if Redis throughput is even a problem in practice. People usually add nodes to get more capacity (memory).

ihsw28y ago

For most use-cases, Redis nodes will saturate network IO before CPU, but it should be noted that Redis is single-threaded so YMMV.

joneholland8y ago

We do heavy pipelining and tend to max out the single thread, so we run multiple redis instances per host.

mangatmodi8y ago

detaro8y ago

They list the two things they changed compared to redis in the readme.

mangatmodi8y ago

1 more reply

ddorian438y ago

I think they are also working on: https://github.com/fastio/pedis (redis implemented on top of seastar-framework / scylladb )

tuyguntn8y ago

any experience using it in production? looks promising results

ddorian438y ago

still missing many features so I don't think it's ready, you should ask on their issue queue

tyingq8y ago

Diffs of what they changed in /src from Redis 4.0: https://github.com/asaradiffs/apsaradiffs

tuananh8y ago

documentation and readme of alibaba projects are scary. Have anyone use this or any alibaba open source projects for production?

CyberDildonics8y ago

Does it still run on only one thread?

rvnx8y ago

This project is some kind of a joke;

Look at the commit history; https://github.com/alibaba/ApsaraCache/commits/develop

15 are "writing of README / adding my copyright", and 2 are actual feature (and not long)

zokier8y ago

I think this is what you are missing: https://github.com/alibaba/ApsaraCache/commit/07518063b17b13...

detaro8y ago

Kind of sad how things can't be partial and in the process of being open-sourced, no they have to be complete or they are "jokes".

dang8y ago

j / k navigate · click thread line to collapse