Comments Locked

57 Comments

Back to Article

  • enoxseven - Tuesday, March 12, 2013 - link

    Are your database servers virtualized as well?
  • Anand Lal Shimpi - Tuesday, March 12, 2013 - link

    Not yet. In order to simplify deployment we went with two boxes, each being a warm spare for the other. Moving forward we will likely virtualize those platforms as well.
  • Adul - Thursday, March 14, 2013 - link

    I would advise against DB virtualization as we have seen in our customer environments a significant performance impact with going virtualized. Even when we pinned this customer to only SSD the difference in performance was significant. I am sure you will be doing some benchmarks before hand.
  • Doby - Saturday, March 16, 2013 - link

    virtualizing can be done so it doesn't have a performance hit. To the contrary, in many situations you can gain performance on large servers by virtualizing. Sure, its not an absolute, but to say virtualization cause significant performance impact is the same as saying a physical server can cause significant performance impact.

    I thought we were getting past the days where people blindly blamed virtualization. That statement should be as dead as saying you should have dedicated RAID10 for DB, and dedicated RAID for logs. It completely depends on the infrastructure.
  • gamoniac - Monday, March 18, 2013 - link

    @Adul, just being curious, in your customer's SQL VM, were the data files/log files residing on the VM disks? A more scalable approach would be to have the OS and SQL software on the VM with data/log files on another physical storage. I am interested in your experience. TIA.
  • kolbryn - Thursday, March 21, 2013 - link

    High disk IO for VMs requires special customizations such as PVSCI Storage Adapter or NPIV. As always disk RAID/type/Meta-LUN are all key to performance for any DB, physical or virtual. Also MB/s is not the best indication of performance, enterprise DBs live or die by IOPS, queue depth and response time.
  • lwatcdr - Thursday, March 14, 2013 - link

    Great writeup. I have been wondering about SSDs for databases makes a lot of sense.
  • DanNeely - Tuesday, March 12, 2013 - link

    Why multiple database engines? MySql and MsSql are both good products; but using the two together on a single site seems odd.
  • martajd - Tuesday, March 12, 2013 - link

    The main site appears to be written in ASP.net which naturally plays nice with MSSQL. The Forums use vBulletin which I believe uses MySQL by default. I imagine they are separate in app code and DB code
  • Anand Lal Shimpi - Tuesday, March 12, 2013 - link

    Correct :)
  • randomlinh - Tuesday, March 12, 2013 - link

    Why not the Intel 710's? Too new?
  • epoon2 - Tuesday, March 12, 2013 - link

    my bet is 520s are cheap now in bulk, and proved to be reliable/stable with the software stack.
  • epoon2 - Tuesday, March 12, 2013 - link

    i meant x25
  • Anand Lal Shimpi - Tuesday, March 12, 2013 - link

    At the time they were too new, but these days I'd go for that and/or the S3700.
  • webmastir - Tuesday, March 12, 2013 - link

    Here come all the posts about how you're doing it wrong.

    Love these posts. Thanks
  • icrf - Tuesday, March 12, 2013 - link

    Doesn't have to be taken as "what you're doing wrong" as much as inquiring "what was your reasoning for this instead of that?"
  • webmastir - Tuesday, March 12, 2013 - link

    True. I guess it's just a matter of how you word it.
  • enoxseven - Tuesday, March 12, 2013 - link

    Looks like they are doing it awesome. Just want some more information.
  • evilspoons - Tuesday, March 12, 2013 - link

    Huh, I'm surprised X25s are available any more. Are they newly manufactured or old stock? I'd be leery about using them simply because they might have been sitting on a shelf for a couple years!
  • Anand Lal Shimpi - Tuesday, March 12, 2013 - link

    These are old stock, we've been on them for over a year now and had them in house a bit before that too. So far everything has been nice and stable thankfully!
  • extide - Tuesday, March 12, 2013 - link

    I would have gone with some form of software RAID vs using Intel RAID. Preferably something like ZFS or MDADM. Even for the MSSQL setup, I would say run a pair of mirrored drives for the OS and then use a ZFS array mounted from another box over iSCSI or fiberchannel or something.
  • extide - Tuesday, March 12, 2013 - link

    To clarify, I am kind of against ALL hardware raid these days, besides using simple on-board RAID for a mirror.

    I generally would setup systems using basic on-board raid with a mirror for all system drives, and then all data drives would use some form of software raid, ZFS preferably, and then those volumes could be mounted up where needed via things like iSCSI or fiberchannel.
  • mfenn - Tuesday, March 12, 2013 - link

    Sounds like whole site in Windows unfortunately. Maybe the next upgrade will be to a real OS? ;)
  • Egg - Tuesday, March 12, 2013 - link

    Where does it say that they use Windows?
    http://www.intel.com/support/chipsets/imsm/sb/cs-0... Intel RAID works on Linux.
  • Gigaplex - Wednesday, March 13, 2013 - link

    It states they use MSSQL, which only runs on Windows.
  • FunBunny2 - Tuesday, March 12, 2013 - link

    Anand: Would you consider measuring the Olde HDD machines against a New SSD Machine, with some (or all) of the tables fully normalized on the SSD, and the flatfile-ish Olde Tables on the Olde Machine? I think that's the truest measure of the value of SSD: largely sequential on HDD versus random/join/synthesize on SSD.
  • johannes - Tuesday, March 12, 2013 - link

    Is the performance/stability/maintenance advantage of dedicated raid-cards compared to softraid so small that softraid is used even in high-traffic servers? Is Intel softraid very different from mdadm-raid in this respect?
  • erple2 - Tuesday, March 12, 2013 - link

    I don't know that the hardware raids have any of those advantages - maybe stability, but they are not faster any more (particularly with SSDs). They are also more of a pain with maintenance - Anand even mentioned in the article that validating new Firmware on them is time consuming and "a pain". These days, I'm not really sure why anyone would go with a hardware RAID device (except MAYBE a giant SAN type operation - but even there, there's probably an underlying problem with how you're approaching the problem that can be re-tooled smarter).

    I think that the days of giant multimillion dollar RAID arrays are slowly going by the wayside, other than support of "decades" old computing platforms that are substantially incorrectly thought to be too expensive to replace.

    I recall having a long talk with someone that suggested that relying on single points of failure (like a giant SAN, for example) is ultimately the wrong direction to go (particularly given the outrageous expense of the hardware - they usually cost more than 2 years of development effort to come up with a clever-er use of smaller, cheaper, more parallel storage arrays).

    However, I'm probably wrong!
  • jmke - Wednesday, March 13, 2013 - link

    it depends; RAID card on PCIe, PCIe bandwith is 64GB/s
    SATA 3.0 spec: 600MB/s

    you do the math :)
  • Gigaplex - Wednesday, March 13, 2013 - link

    So what if PCIe bandwidth is higher than SATA? You've still got the SATA limitation from the drive to the RAID card in the first place. Unless you've got something exotic like fibre channel. But then why are you comparing to a single SATA port?
  • Firedron - Wednesday, March 13, 2013 - link

    You’re not right – with large number of drives and high cpu load hardware RAID will be always faster. ASICs are always better than general purpose CPUs.
    And it really depends on your server usage scenario. If you’re using software RAID, you’re taking resources from CPU – for RAID1/10 it may be not very much, because you’re not calculating parity. But with large number of drives, especially SSDs, and RAID5/6 – it’s a lot of CPU cycles/bandwidth penalty, which can be used by application. In that case, it is really worth to invest into hardware raid controller. And if you’re buying server from big USA vendor with vendor’s raid card - it will be under warranty – firmware issues is not your problem – vendor will test it hundred times with all configurations before releasing public. That why you are paying premium buying enterprise level hardware from premium vendor.
  • drinking12many - Thursday, March 14, 2013 - link

    I dunno have you looked at something like an enterprise Software Raid type setup. Most SANs such as Netapp and Oracle dont really have raid cards. They are just glorified FreeBSD or Solaris installs and depend heavily on the CPU to do the parity work. On dedicated SAN boxes who cares how much cpu the Filer is using long as its handling your workload. We just bought some Oracle ZFS appliances using 4x8core CPUS enough for parity, compression, checksums and maybe even some dedupe if we want. I would agree if your doing the storage on box that a raid controller is probably the way to go but on the SAN side its usually better without since thats the only workload they have to run.
  • mike55 - Tuesday, March 12, 2013 - link

    Anand, what's the largest amount of writes you've seen on one of your SSDs? Have you ever seen one fail due to old age or start losing capacity due to sectors failing (I'm not sure how SSDs handle losing lots of NAND)?
  • andy318 - Tuesday, March 12, 2013 - link

    In your new setup with software RAID, do you have write caching turned on? If so, is data in the cache protected from power loss?
  • ibb_1976 - Tuesday, March 12, 2013 - link

    Anand, can you include tests that where presented on Usenix this year (https://www.usenix.org/conference/fast13/understan... in your SSD reviews. I think these guys found some important things regarding failures, but they not provide info about models and manufacturers of tested SSD. Thank you!
    PS. Sorry about my english, it's not so good.
  • ibb_1976 - Tuesday, March 12, 2013 - link

    https://www.usenix.org/conference/fast13/understan...
    Sorry, this is the right link.
  • sideshow23bob - Tuesday, March 12, 2013 - link

    Do you expect you'll have to replace any of your SSDs before the next hardware change (due to hitting write cycle limit? Feel free to explain much more about your thoughts on that or keep vague if this is of strategic importance to you or the site. Thanks and looking forward to reading many more articles with the nice, new design.
  • bobbozzo - Wednesday, March 13, 2013 - link

    ASP.net is normally run on Windows.
  • bobbozzo - Wednesday, March 13, 2013 - link

    This above was in reply to another post. I'm not sure the threading/nesting is working right.
  • jmke - Wednesday, March 13, 2013 - link

    >> "The reliability side was simple to understand - with a good SSD, I wouldn't have to worry about my drive dying unexpectedly. Living in fear of a testbed hard drive dying over the weekend before a big launch was a thing of the past. "

    How are SSD failing unexpectedly any different from HDD? In fact they are worse. If a HDD starts failing, you most likely get a chance to do some data recovery, even restore it back to working order if there only failing sections.
    With an SSD it's light out. reboot device not found kind of fun. I've had 2 Intel SSD fail out of a batch of 30 in less than 3 months... so I would say that MTBF for SSD is not much different from HDD imho.
    What might be different is environmental impact on failure causes, SSDs can take more physical abuse, and with testbed PCs they would be more error-resistant I guess :)
  • canthearu - Wednesday, March 13, 2013 - link

    You are entirely wrong with your approach to hard drives.

    In the world of IT, you start having any trouble with any hard drive or SSD, you pull it and replace it right away, and you either rebuild the array, or you restore from backups.

    The cost in time, and the risk of failure, means that you generally don't attempt to restore a hard drive to working condition, or recover data from a storage device, as anything except a very last resort. If you are forced into doing this, it also means your IT procedures and policies have failed.
  • nexox - Wednesday, March 13, 2013 - link

    With well-tested SSDs, running validated firmware, in a proper environment, you're going to see very, very few unexpected failures. Most surprise SSD failures are caused by firmware bugs that are generally tickled by low-quality disk controllers and/or unexpected power loss. If you have high-quality SAS controllers (not necessarily a hardware RAID adapter) and you run plenty of power failure tests against a drive, you can be fairly sure it will last you quite a long time in production.

    And physical durability is actually important - high-density servers that run many spinning drives typically have vibration issues that can significantly shorten drive life span. SSDs are unaffected by vibration, and they don't cause more vibration that can affect any other spinning disks that you may have in your chassis / rack.
  • Michael McNamara - Wednesday, March 13, 2013 - link

    Those performance numbers are very impressive... how are you backing up all that data? I'm guessing you have a disk to disk solution.

    Cheers!
  • Hrel - Wednesday, March 13, 2013 - link

    You talk about going to SSD's but what about storage space? You said you use 6 SSD's but even that's not much space. What do you to store data? SSD's still aren't any good for that.
  • brshoemak - Wednesday, March 13, 2013 - link

    The previous article that detailed the new hardware indicated that the image hosting duties, were being shifted to a cloud computing company. Images and multimedia usually account for a sizable chunk of a website. The remaining content would not take up THAT much space by comparison.

    I would be interested to know how much data makes up the main site versus the amount of data for the forums.
  • andrewbuchanan - Wednesday, March 13, 2013 - link

    Did you choose asp.net because of ease of development, output caching, something else? How do you find Windows vs Linux from a reliability standpoint?

    I have a mixture of both at work and have for the most part been happy with both. Lots of forums bash windows web servers but I've found them to work very well (no worse than lamp) so I'm curious how you're experience has been.
  • InternetGeek - Thursday, March 14, 2013 - link

    Anand,

    Do you think you would see any benefits for moving in a cloud infrastructure?. Azure recently started offering full SSD environments, and I believe it would be simple to migrate to such a set up.

    What do you think could be the trade-offs?
  • Dradien - Friday, March 15, 2013 - link

    I've been coming to this site since I think 2002 or so. Consistently one of the best sites on the internet. Going beyond which videocard/CPU/SSD/etc etc is better, you seem to delve deep into the how's and why's stuff perform like they do, and it's nice to see that.

    That and you seem like a chill guy Anand, talking to your commenters and audience, seems like you honestly love and care about what you do, and have a passion for it, and I can respect that.
  • rschmitty - Wednesday, March 20, 2013 - link

    "we settled on 160GB Intel X25-M G2s but partitioned the drives down to 120GB in order to ensure they'd have a very long lifespan"

    Could someone elaborate on that? How does partitioning ssd give you a longer lifespan?
  • Hrobertgar - Wednesday, March 20, 2013 - link

    I assumed the word 'partition' referred to having one 120GB 'partition' and 40GB reserve, something AnandTech has mentioned in many SSD reviews (I think usually in terms of I/O consistency); as oppopsed to having one 120GB partition and a second 40GB partition.
  • Falkinator - Wednesday, March 27, 2013 - link

    Over provisioning allows the drive to spread out its data and optimize where data gets written in order to reduce the write cycles on a particular cell. I just learned about this setting up my Samsung 840 Pro. Samsung recommended 7-10 percent.
  • tejas@atech - Thursday, March 21, 2013 - link

    @anand Sorry to even ask this,But in post u wrote "we settled on 160GB Intel X25-M G2s but partitioned the drives down to 120GB in order to ensure they'd have a very long lifespan."
    I also use it , although not on par with R/W scale. But how can this increase Lifespan of SSD? :-0
  • gospadin - Friday, March 22, 2013 - link

    Some SSDs, when you don't write to the entire drive, will use the extra space for their internal management automatically, which may lower write amplification. Lower write amplification generally translates to better performance as a first order effect, or if you run the same workload with less write amp, you'll induce fewer cycles on the media and thus take longer to hit the media's endurance limits. As such, this will increase the lifespan of the drive.
  • wang2013 - Tuesday, March 26, 2013 - link



    服を購入する場所を見つけると、一つは、彼らが着たいものを決定する必要があります。冬が終わるとして、夏前には、誰もが幸せな色合いを探します。それが今は少し遠いように思えるかもしれま<a href="http://www.nikezu.com/B9-c-297.html">サングラス サングラス スニーカー</a>
    せんが、たった一ヶ月かそこらでこの変更を探します。あなたは卸売衣類を購入している場合は、これらの明るく元気なトーンを考える。彼らはいつもイースターヶ月ほど適しています!その他の傾向としては、アクティブな摩耗の衣類のインスピレーションは、春のスタイルになることは間違いありません。これらのインスピレーションピースは古いに若いから、すべての年齢層で利用できるようになります。キーは快適であるものです。とき卸売衣服ショッピングあなたは快適、スポーツスタイルの選択で間違った方向になることは決してありません。あなただけのスタイルに何があるか<a href="http://www.nikeinjapan.com/air-jordan-c-55.html&qu...
    わからない場合は、有名人が公共の場で着ているものを見つけるためにいくつかの研究を行う。金持ちや有名にしている人々は通常、 "内"で衣類の種類を決定し、この春には、ルールの例外ではありません。人気のファッションや娯楽雑誌を拾い読みすると、また、あなたの現在の人気の傾向についてのヒントをたくさん与える。また、あなたの買い物は服を販売している偉大なwardrobe.Anyoneを意味することができます前に、格好悪い(あなたが知らない、念のため。)これらのヒントをお探しているものの絵を与えるために含まれていないことに注意を払うべきであることは十分承知している在庫がファッションの方法によって決定されるという事実。 1シーズンからホットな<a href="http://www.sunkutu.com/new-balance-cm999lgy-cm999l... Balance CM999LGY ニューバランス CM999LGY</a>
    服は、来シーズンの不発弾かもしれません。したがって、それは知っていることが不可欠であり、商品を販売するためには、ファッションのトレンドを追う。ありがたいことに、それはファッショナブルな卸売服を見つけることはとても簡単です。あなたが卸婦人服に特化卸売衣料品の出品がありますインチ卸売紳士服に特化した人も、興味があるかもしれないこと衣類のほとんどあらゆるタイプの問屋があります。子供服、アクセサリーや靴に特化した他の人に特化した他はまだあります。これらのオプションのいずれかが良い購入のために作ることができます。卸売衣服のための信頼できるソースを見つけるためにオンラインで検索します。多くのウェブサイトでは、卸売小売業者のディレクトリを使用しています。衣類の販売を含め、すべての別のカテゴリの卸売ディレクトリリスト卸売業者、。卸売ディレクトリを通して、あなたは購入可能である商品のほぼすべてのタイプを見つけることができます。あなたが使用したいの卸売衣服サイトを見つけたら、それを購入する前にチェックアウトするいくつかの時間がかかる。彼らのリストを追加することを可能にする前に、ウェブサイトをチェックアウトするディレクトリがありますが、リスト上の制限がありません他の人が存在します。
  • wang2013 - Tuesday, April 9, 2013 - link

    <P>エアジョーダンでしょう。上のエアジョーダンの靴で、1つは常にすべての時間を良い感じになります。<a href="http://www.worldsportsbrand.com/サッカーユニフォームナショナルチーム...、注意してください。あなたはナイキヨルダンの靴のいずれかを着用するときは、インスタント有名人になるため:すべてのショットは慎<a href="http://www.airjordanba.com/air-force-c-943.html&qu... force 1</a>重に見守っている、すべての動きが観客とあなたが法廷の内外で行うすべての喜びですが、人々のための画像と動画でキャプチャされますごちそうに。エアジョーダンの靴は、すべてのバスケットボール選手に不可欠であるためです。それ以外の場合は、これまでバスケットボールハードコートに足を設定する最大の選手にちなんで命名されませ<a href="<a href="http://www.airjordankuz.com">ナイキ エア</a>ん。デニス·ロッドマンは、かつてマイケルジョーダンは彼の私生活で、彼はバスケットボールを果たしている方法で、両方の大きな影響であると述べた。ワームは、彼はすべてのゲームで、様々な方法で染色された彼の荒いゲーム、ピアス、カラフルな髪のために知られていた彼はNBAで足を踏み入れたとき、彼が最初に知られていた。<a href="http://www.glassesjapan.com/oakley-BC-c-1098.html&... オークリー ワイヤー OAKLEY オークリー ワイヤー スニーカー</a>彼は権限を持ついくつかの選手や人々とランイン多数の手抜きと、いくつかのもつれになったとして、裁判所の外では、ロッドマンは同然であった。それから彼は、スコッティーピッペンと一緒にヨルダンの副官の1になるためにシカゴ·ブルズに移籍した彼の決定的な瞬間が来た。</p>
  • wang2013 - Friday, April 26, 2013 - link

    別の男女兼用のお気に入り人気シチズン 腕時計は実際RB2079です。ミニマリズムのインスタンスは、このスムーズな金属フレームの眼鏡は、柔軟な依存·と実際にセミリムレスです。寺院はそれにエッチング実際のブランドロゴ 'RB'が。 'これらの眼鏡は、贅沢な素材、力強いスタイル、精密なディテールの混合物である。鮮度と品質は、あなたが選択した人気Evisu ショートジーンズ任意のレイバンの眼鏡と一緒に取得します。彼らはすべてのイベントに理想的な摩耗のために作る。ホルマール、カジュアル、上に置くラウンジ、それをチェックするために常に準備レイバンがある、あなたのスタイルが何であれパーティーウェア。レイバンのサングラスは、太陽の損傷Evisu ジーンズだけでなく、長い目を通して、彼らが健康で強い両方維持に対する適切な保護機能を有効にしてアマガエルを提供することが保証されています。レイバンのサングラスのペアを購入することは間違いなく賢明です。レイバンの色合いは、さまざまなリーグになりたい人のためのものです。prada サングラスあなたのペアを購入することを決定した場合はそのため、あなたは簡単にブランドの本物のデザイナーサングラスの最寄りの店舗の中から注文する、または1つをオンラインで購入することができます。
  • wang2013 - Friday, April 26, 2013 - link



    あなたは最高の腕時計をしhttp://www.glassesjpsale.com">カレラサングラスたい場合は、どこか別の場所に見える。あなたは最高のmp3プレーヤーにしたい場合は、iPodを試してみてください。両方のビットをしたい場合でも、カシオWMP-1VのMP3はあなたのためです。あなたが約4倍いつものサイズで腕時計を身に着けている気にしない場合はそのためです。あなたが他の野外活動を実行し、実行するためのセキュアなMP3プレーヤーをしたい場合は、この時計は素晴らしい作品です。http://www.watchjpshop.com">人気カジュアル腕時計それは間違いなく、あなたはそれがあなたの手首に縛り付けているに一度落ちするつもりはない。しかし、その大きさのために、私は本当に感銘を受けていることだけではありませんでした。ボタンがナビゲートする少し楽だったらおそらくこの時計は、それが少しビット小さかった場合にも、私にはより魅力的になり、より自然な方法で自分の手首にフィットするだろう。観察する現在の時刻を表示小さなポータブルクロックです。カシオでは、東京に本社を置く電子ユニットの生産事業である。http://www.evisujpsale.com/evisu-ショートジーンズ-c-1.html">Evisu ショートジーンズ事業は忠雄鹿塩として認識製造技術に特化したエンジニアが、4月に12ヶ月、1946年に設立されました。彼らは時計が、他の製品およびソリューションの開発を始めていなかった開始からカシオ株式会社。彼らの主要な製品やサービスに開始するも、指が完全にそれを無料でスポーツ男または女を残しているが、着用者は、その要点にタバコを吸うようにダウンさせる、http://www.evisujpshop.com">エヴィス ジーンズ指輪パイプ、タバコを開催する指輪だった。離れて時計から他のカシオソリューションは、鍵盤楽器、電卓、デジタルカメラ、ノートパソコン、携帯電話、ドル·レジスタ、電子辞書、デジタル日記他の個人の中にある。今日では、カシオは、ほとんど正常に弾力性、相当な大きさの時計のために識別されます。

Log in

Don't have an account? Sign up now