Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [..X...X........X.....X.X............X......X.X.XXXX.X.XX.X.X] duration: 5.00060682s trace: https://jaeger.dev.storj.io/trace/753258dd60ddefa3 FAIL (2025-12-30 10:46:48 UTC) orbiter warning: context deadline exceeded Duration: 5.000628151s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XX.X.X.X.XX.X.....XX..........XX...X.X..XX.XX........X......] duration: 6.495211704s trace: https://jaeger.dev.storj.io/trace/5c4f8766c61bbc8a FAIL (2025-12-30 10:46:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: 6OIBW7HMNQR3UXQYWFDP6J2AZDSL35VO4Z6PJ5P7UCRIJFALL77A) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: KVNNUQGW7FKC5F77RAFEFUFSMXZ62FT7RIVO5F6JFZBCGD7YC4TA) context deadline exceeded ======LOGS====== 0 10:41:37.067033916 (595h12m36.938116961s) [upload=521763]: Starting upload Dec 30 10:41:37.067118447 (595h12m36.938201492s) [upload=521763]: Done starting upload Dec 30 10:41:37.067361941 (595h12m36.938444976s) [upload=521763]: Uploading segments... Dec 30 10:41:37.067504051 (595h12m36.938587096s) [upload=521763]: Got next segment. Inline: false Dec 30 10:41:37.067575377 (595h12m36.938658412s) [upload=521763]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:41:37.067583141 (595h12m36.938666186s) [upload=521763]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:41:37.315237828 (595h12m37.186320863s) [seg_pos={0 0}, upload=521763]: Begin upload segment... Dec 30 10:41:37.315281842 (595h12m37.186364877s) [seg_pos={0 0}, upload=521763]: Done begin upload segment. Dec 30 10:41:37.315306599 (595h12m37.186389644s) [noise=true , address=77.90.31.13:4629 , piece_id=SNJ4Z5D4QNHBSNDLSH7W6NROV3X3YGQYB2MNAIKURRUBSTCNIGOA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.315384688 (595h12m37.186467723s) [noise=true , address=145.79.227.20:4439 , piece_id=QKDITA4DSTKAIU2S2JXG5H7JM5FH2JE7CDUHAMLMYVPEGF53VOSQ, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.315849714 (595h12m37.186932759s) [noise=true , address=145.79.227.25:4619 , piece_id=QPAQFNMX4S7OY5YYUXQQKUG7UYHCXBDSKBL6A7S45OQTY6BYWTNA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.316276256 (595h12m37.187359291s) [noise=true , address=77.90.31.16:4735 , piece_id=KVNNUQGW7FKC5F77RAFEFUFSMXZ62FT7RIVO5F6JFZBCGD7YC4TA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.316667430 (595h12m37.187750476s) [noise=true , address=145.79.227.48:4303 , piece_id=CXP7CRYIP4NUZYMTYLRA4N4YLZHSOFOK5B5RF7JXCCAQMTX2GHLA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.317060378 (595h12m37.188143434s) [noise=true , address=145.79.227.19:4077 , piece_id=4CRE76JFUMHITF7OXCAIC455F7L5SJC5ZCDMDSIJJ6AQ4HSK7AMA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.317419994 (595h12m37.188503029s) [noise=true , address=45.91.116.12:4225 , piece_id=7H55Y5PZTFNLY4SNJDXD2PEARFNP6OXRJJGXRDA2UMPM435B4K3A, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.317948861 (595h12m37.189031896s) [noise=true , address=145.79.227.39:4014 , piece_id=Q7CWPW2G3K2O5NZVZQTLZAXJRZG5XYFZZJNP4SHVEIH6RW6CAUOA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318277436 (595h12m37.189360481s) [noise=true , address=145.79.227.17:4970 , piece_id=VFVP5IJDEG4OUH6H6QHDCKNORBC5US4ZG3273FPBFT7BO3DNYNMA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318313034 (595h12m37.189396069s) [noise=true , address=145.79.227.45:4229 , piece_id=ZNC76TIUYUXEMZC5FY22U365CICRNFQCSLIXPVZDBF2KNZ2KDI7Q, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318343201 (595h12m37.189426236s) [noise=true , address=145.79.227.21:4475 , piece_id=WP7NRZXGIFRCI3JVCVCXTVXEFZ7BYZQN5JXLTREWXSKD6ULZYSVQ, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318616161 (595h12m37.189699196s) [noise=true , address=145.79.227.11:4730 , piece_id=QMESJWSNK2KAIDZBQWU2OUNMSPEZFWGQXU3QVNFBVEQXYWHPLVXA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318826802 (595h12m37.189909837s) [noise=true , address=145.79.227.50:4374 , piece_id=G5Y7BV34YTPBC7QK3A5OVO3P7ILJT5WYOOXZBAA6QJXPXVCK5UXQ, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318975465 (595h12m37.19005851s) [noise=true , address=145.79.227.31:4799 , piece_id=JEBR7PFHJBN4UOTVN5HM3NNZ5WX4IWZNBRUFFJ3QSDSTONAFAR7Q, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.319245860 (595h12m37.190328895s) [noise=true , address=77.90.31.11:4554 , piece_id=UUJNARWXIOJM775GZKLOXT4ZHBUYDHVTI5STPK2Z7UHYSTGIOV3A, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.321923038 (595h12m37.193006083s) [noise=true , address=77.90.31.18:4808 , piece_id=6OIBW7HMNQR3UXQYWFDP6J2AZDSL35VO4Z6PJ5P7UCRIJFALL77A, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:38.919914306 (595h12m38.790997351s) [upload=521763]: Next returned nil segment Dec 30 10:41:38.919944133 (595h12m38.791027178s) [upload=521763]: Waiting for error group managing segments... Dec 30 10:41:39.299400313 (595h12m39.170483348s) [noise=true , address=145.79.227.31:4799 , piece_id=JEBR7PFHJBN4UOTVN5HM3NNZ5WX4IWZNBRUFFJ3QSDSTONAFAR7Q, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.301898620 (595h12m39.172981665s) [noise=true , address=145.79.227.21:4475 , piece_id=WP7NRZXGIFRCI3JVCVCXTVXEFZ7BYZQN5JXLTREWXSKD6ULZYSVQ, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.305326327 (595h12m39.176409362s) [noise=true , address=145.79.227.50:4374 , piece_id=G5Y7BV34YTPBC7QK3A5OVO3P7ILJT5WYOOXZBAA6QJXPXVCK5UXQ, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.305621429 (595h12m39.176704474s) [noise=true , address=145.79.227.45:4229 , piece_id=ZNC76TIUYUXEMZC5FY22U365CICRNFQCSLIXPVZDBF2KNZ2KDI7Q, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.313049473 (595h12m39.184132508s) [noise=true , address=145.79.227.48:4303 , piece_id=CXP7CRYIP4NUZYMTYLRA4N4YLZHSOFOK5B5RF7JXCCAQMTX2GHLA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.313096753 (595h12m39.184179798s) [noise=true , address=145.79.227.25:4619 , piece_id=QPAQFNMX4S7OY5YYUXQQKUG7UYHCXBDSKBL6A7S45OQTY6BYWTNA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.330307055 (595h12m39.20139009s) [noise=true , address=145.79.227.17:4970 , piece_id=VFVP5IJDEG4OUH6H6QHDCKNORBC5US4ZG3273FPBFT7BO3DNYNMA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.331102860 (595h12m39.202185895s) [noise=true , address=145.79.227.11:4730 , piece_id=QMESJWSNK2KAIDZBQWU2OUNMSPEZFWGQXU3QVNFBVEQXYWHPLVXA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.333242113 (595h12m39.204325158s) [noise=true , address=145.79.227.19:4077 , piece_id=4CRE76JFUMHITF7OXCAIC455F7L5SJC5ZCDMDSIJJ6AQ4HSK7AMA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.355266472 (595h12m39.226349507s) [noise=true , address=145.79.227.20:4439 , piece_id=QKDITA4DSTKAIU2S2JXG5H7JM5FH2JE7CDUHAMLMYVPEGF53VOSQ, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.387551277 (595h12m39.258634312s) [noise=true , address=145.79.227.39:4014 , piece_id=Q7CWPW2G3K2O5NZVZQTLZAXJRZG5XYFZZJNP4SHVEIH6RW6CAUOA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.876119214 (595h12m39.747202249s) [noise=true , address=77.90.31.16:4735 , piece_id=KVNNUQGW7FKC5F77RAFEFUFSMXZ62FT7RIVO5F6JFZBCGD7YC4TA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:40.191523548 (595h12m40.062606593s) [noise=true , address=77.90.31.18:4808 , piece_id=6OIBW7HMNQR3UXQYWFDP6J2AZDSL35VO4Z6PJ5P7UCRIJFALL77A, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:40.191549366 (595h12m40.062632401s) [seg_pos={0 0}, upload=521763]: Segment reached optimal threshold of 13 pieces. Dec 30 10:41:40.191708269 (595h12m40.062791304s) [noise=true , address=77.90.31.11:4554 , piece_id=UUJNARWXIOJM775GZKLOXT4ZHBUYDHVTI5STPK2Z7UHYSTGIOV3A, seg_pos={0 0}, upload=521763]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:35868->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:35868->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35868->77.90.31.11:4554: use of closed network connection Dec 30 10:41:40.191714180 (595h12m40.062797225s) [noise=true , address=77.90.31.11:4554 , piece_id=UUJNARWXIOJM775GZKLOXT4ZHBUYDHVTI5STPK2Z7UHYSTGIOV3A, seg_pos={0 0}, upload=521763]: Long tail cancellation due to optimal threshold Dec 30 10:41:40.191795394 (595h12m40.062878459s) [noise=true , address=77.90.31.13:4629 , piece_id=SNJ4Z5D4QNHBSNDLSH7W6NROV3X3YGQYB2MNAIKURRUBSTCNIGOA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 30 10:41:40.191814181 (595h12m40.062897236s) [noise=true , address=77.90.31.13:4629 , piece_id=SNJ4Z5D4QNHBSNDLSH7W6NROV3X3YGQYB2MNAIKURRUBSTCNIGOA, seg_pos={0 0}, upload=521763]: Long tail cancellation due to optimal threshold Dec 30 10:41:40.191917547 (595h12m40.063000583s) [noise=true , address=45.91.116.12:4225 , piece_id=7H55Y5PZTFNLY4SNJDXD2PEARFNP6OXRJJGXRDA2UMPM435B4K3A, seg_pos={0 0}, upload=521763]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 30 10:41:40.191923709 (595h12m40.063006744s) [noise=true , address=45.91.116.12:4225 , piece_id=7H55Y5PZTFNLY4SNJDXD2PEARFNP6OXRJJGXRDA2UMPM435B4K3A, seg_pos={0 0}, upload=521763]: Long tail cancellation due to optimal threshold Dec 30 10:41:40.191939890 (595h12m40.063022935s) [seg_pos={0 0}, upload=521763]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:41:40.192001908 (595h12m40.063084943s) [upload=521763]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:41:40.192006397 (595h12m40.063089442s) [upload=521763]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:41:40.486537746 (595h12m40.357620801s) [upload=521763]: Done uploading segments... Dec 30 10:41:40.486564095 (595h12m40.35764715s) [upload=521763]: Upload finished. err: <nil> Duration: 6.495237933s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................X....X.........X....................X] duration: 1m0.003682404s trace: https://jaeger.dev.storj.io/trace/427bb74fbf2823be FAIL (2025-12-30 10:46:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.44:4223 , piece_id=CKJWI5GBSAX4RYMDNVDXWDC2OYCV5XQZ7QIGL6IVGFXHHIDT3HSQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.712977061 (595h16m21.584060096s) [noise=true , address=77.90.31.14:4693 , piece_id=3T7WRF3NCBPKWB66GWNZH3ABQZMA6HKYKICXZ7U2D7TJFQS5BPPA, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.713064618 (595h16m21.584147663s) [noise=true , address=145.79.227.40:4071 , piece_id=DQBKG2VXAJLJJKRJGNJZZHIQYMDEMBY6PEUJM3IZDA2L6L5GPPYQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.713493665 (595h16m21.5845767s) [noise=true , address=145.79.227.2:4146 , piece_id=G7G6ETKD5ES7VURFWK63BC6UX27DWNUEBQNQRR3WMULN6QQITARQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.713746215 (595h16m21.58482925s) [noise=true , address=193.149.164.143:4698 , piece_id=2ZLC6LPMSB6DEOE7SZJ7ZO36NESW5BD5WQPRJV3KERNCTMJOCUAA, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.713961054 (595h16m21.585044099s) [noise=true , address=77.90.31.13:4642 , piece_id=VSI2U2IC2334DFBMJ4Y4VYK256AJUXJ4VS2URZ55R2Z4U3F4ONNQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.714248462 (595h16m21.585331507s) [noise=true , address=77.90.31.16:4760 , piece_id=5AZVT6BGTIKWITE2ROYT5RJKMS6N34PW7NT4CZAATSEU7EP52C4Q, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.714290352 (595h16m21.585373387s) [noise=true , address=77.90.31.19:4865 , piece_id=AZCHTN5JRY2QGAECWZZKF5RM5N65W67JOEHPPXGFBLDMCACLJYRA, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.714513316 (595h16m21.585596361s) [noise=true , address=145.79.227.5:4249 , piece_id=64QOVSR6OHSC42N2PRUWESB4IO24ILEVAVFCTS4B5XIMV235EQVQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.714907127 (595h16m21.585990172s) [noise=true , address=145.79.227.43:4173 , piece_id=HEUCL4A4EPNDH4CNII5NSY4PHVX7PJP3LESIWBDJ4KJI4N44Z6KA, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.715511768 (595h16m21.586594813s) [noise=true , address=145.79.227.30:4765 , piece_id=XZWZYWKFLBQJJDZONDBCPWBYITBL7JDOAMIZEZAIIMR2LPPMZOVQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.716418374 (595h16m21.587501419s) [noise=true , address=64.49.13.169:4706 , piece_id=QLHYYGVTGE5LE3L7EAPNVDC74262GJIJA67NCTY54FNKLMKVHQNA, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.718018150 (595h16m21.589101185s) [noise=true , address=145.79.227.49:4364 , piece_id=J7I2XV5WQ2XHGSZJUM3GO7UYVPS6S2QWUGWT2YKITCFPV7QPFDNQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.718481222 (595h16m21.589564257s) [noise=true , address=145.79.227.4:4210 , piece_id=SMS65JPDA5DVT37W6VAJPPMYH6QKFFUYN36C73TZPKIS7MMRXKUA, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.719042791 (595h16m21.590125836s) [noise=true , address=77.90.31.11:4562 , piece_id=B67SGHB6QI6RFW52PAEQ2KKRZQFDYOZYRI7NNSMUNCNUBSA3UXXA, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:21.719436842 (595h16m21.590519887s) [noise=true , address=145.79.227.33:4892 , piece_id=POYOGWTGKWQXQRYSI6OZZNYZMKPOCUTWXGEWS4CEIYR2V2WKGRPQ, seg_pos={0 0}, upload=521821]: Uploading piece... Dec 30 10:45:23.159616720 (595h16m23.030699765s) [upload=521821]: Next returned nil segment Dec 30 10:45:23.159642560 (595h16m23.030725605s) [upload=521821]: Waiting for error group managing segments... Dec 30 10:45:23.568018017 (595h16m23.439101062s) [noise=true , address=145.79.227.2:4146 , piece_id=G7G6ETKD5ES7VURFWK63BC6UX27DWNUEBQNQRR3WMULN6QQITARQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.583184015 (595h16m23.45426706s) [noise=true , address=145.79.227.5:4249 , piece_id=64QOVSR6OHSC42N2PRUWESB4IO24ILEVAVFCTS4B5XIMV235EQVQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.586325036 (595h16m23.457408081s) [noise=true , address=145.79.227.4:4210 , piece_id=SMS65JPDA5DVT37W6VAJPPMYH6QKFFUYN36C73TZPKIS7MMRXKUA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.806152585 (595h16m23.67723563s) [noise=true , address=77.90.31.13:4642 , piece_id=VSI2U2IC2334DFBMJ4Y4VYK256AJUXJ4VS2URZ55R2Z4U3F4ONNQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.819009537 (595h16m23.690092582s) [noise=true , address=77.90.31.16:4760 , piece_id=5AZVT6BGTIKWITE2ROYT5RJKMS6N34PW7NT4CZAATSEU7EP52C4Q, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.845285812 (595h16m23.716368857s) [noise=true , address=77.90.31.14:4693 , piece_id=3T7WRF3NCBPKWB66GWNZH3ABQZMA6HKYKICXZ7U2D7TJFQS5BPPA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.847447017 (595h16m23.718530062s) [noise=true , address=77.90.31.11:4562 , piece_id=B67SGHB6QI6RFW52PAEQ2KKRZQFDYOZYRI7NNSMUNCNUBSA3UXXA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.856608672 (595h16m23.727691707s) [noise=true , address=77.90.31.19:4865 , piece_id=AZCHTN5JRY2QGAECWZZKF5RM5N65W67JOEHPPXGFBLDMCACLJYRA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.971619162 (595h16m23.842702197s) [noise=true , address=145.79.227.43:4173 , piece_id=HEUCL4A4EPNDH4CNII5NSY4PHVX7PJP3LESIWBDJ4KJI4N44Z6KA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:23.974330555 (595h16m23.84541363s) [noise=true , address=145.79.227.18:4009 , piece_id=74G4RMIC7XOUQII3FVQYAVNIF2AKTQWTK4IEGNEZE7UDBFGQ4WPA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:24.027797295 (595h16m23.89888033s) [noise=true , address=45.91.116.4:4959 , piece_id=2NO7XKBMQASIHCJXHEJAURKY63F7TX3DCNXOPDIQLA5QUJW4MAXQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:24.429331223 (595h16m24.300414258s) [noise=true , address=145.79.227.33:4892 , piece_id=POYOGWTGKWQXQRYSI6OZZNYZMKPOCUTWXGEWS4CEIYR2V2WKGRPQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:24.477331123 (595h16m24.348414178s) [noise=true , address=145.79.227.40:4071 , piece_id=DQBKG2VXAJLJJKRJGNJZZHIQYMDEMBY6PEUJM3IZDA2L6L5GPPYQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:24.545180822 (595h16m24.416263857s) [noise=true , address=145.79.227.44:4223 , piece_id=CKJWI5GBSAX4RYMDNVDXWDC2OYCV5XQZ7QIGL6IVGFXHHIDT3HSQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: <nil> Dec 30 10:45:24.545205568 (595h16m24.416288603s) [seg_pos={0 0}, upload=521821]: Segment reached optimal threshold of 14 pieces. Dec 30 10:45:24.545446567 (595h16m24.416529622s) [noise=true , address=193.149.164.143:4698 , piece_id=2ZLC6LPMSB6DEOE7SZJ7ZO36NESW5BD5WQPRJV3KERNCTMJOCUAA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF Dec 30 10:45:24.545455103 (595h16m24.416538138s) [noise=true , address=193.149.164.143:4698 , piece_id=2ZLC6LPMSB6DEOE7SZJ7ZO36NESW5BD5WQPRJV3KERNCTMJOCUAA, seg_pos={0 0}, upload=521821]: Long tail cancellation due to optimal threshold Dec 30 10:45:24.545641889 (595h16m24.416724935s) [noise=true , address=145.79.227.49:4364 , piece_id=J7I2XV5WQ2XHGSZJUM3GO7UYVPS6S2QWUGWT2YKITCFPV7QPFDNQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12S9DQZhV6SDULbA9WKndh12CpFih6xtNcpdwUApD9d2XThFdFk): protocol: expected piece hash; context canceled; EOF Dec 30 10:45:24.545651768 (595h16m24.416734813s) [noise=true , address=145.79.227.49:4364 , piece_id=J7I2XV5WQ2XHGSZJUM3GO7UYVPS6S2QWUGWT2YKITCFPV7QPFDNQ, seg_pos={0 0}, upload=521821]: Long tail cancellation due to optimal threshold Dec 30 10:45:24.545678299 (595h16m24.416761344s) [noise=true , address=145.79.227.30:4765 , piece_id=XZWZYWKFLBQJJDZONDBCPWBYITBL7JDOAMIZEZAIIMR2LPPMZOVQ, seg_pos={0 0}, upload=521821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT): protocol: expected piece hash; context canceled; EOF Dec 30 10:45:24.545687706 (595h16m24.416770751s) [noise=true , address=145.79.227.30:4765 , piece_id=XZWZYWKFLBQJJDZONDBCPWBYITBL7JDOAMIZEZAIIMR2LPPMZOVQ, seg_pos={0 0}, upload=521821]: Long tail cancellation due to optimal threshold Dec 30 10:45:24.545858341 (595h16m24.416941386s) [noise=true , address=64.49.13.169:4706 , piece_id=QLHYYGVTGE5LE3L7EAPNVDC74262GJIJA67NCTY54FNKLMKVHQNA, seg_pos={0 0}, upload=521821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:49688->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:49688->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49688->64.49.13.169:4706: use of closed network connection Dec 30 10:45:24.545866877 (595h16m24.416949912s) [noise=true , address=64.49.13.169:4706 , piece_id=QLHYYGVTGE5LE3L7EAPNVDC74262GJIJA67NCTY54FNKLMKVHQNA, seg_pos={0 0}, upload=521821]: Long tail cancellation due to optimal threshold Dec 30 10:45:24.545980965 (595h16m24.41706401s) [seg_pos={0 0}, upload=521821]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:45:24.546075204 (595h16m24.417158239s) [upload=521821]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:45:24.546084552 (595h16m24.417167587s) [upload=521821]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:45:24.833013734 (595h16m24.704096779s) [upload=521821]: Done uploading segments... Dec 30 10:45:24.833037870 (595h16m24.704120915s) [upload=521821]: Upload finished. err: <nil> Duration: 1m0.003702853s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [...X...X........X.....X.X............X......X.X.XXXX.X.XX.X.] duration: 4.700531695s trace: https://jaeger.dev.storj.io/trace/38eec761c45b0506 FAIL (2025-12-30 10:46:19 UTC) orbiter warning: context deadline exceeded Duration: 4.70056058s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [X...X...X........X.....X.X............X......X.X.XXXX.X.XX.X] duration: 5.000444812s trace: https://jaeger.dev.storj.io/trace/38eec761c45b0506 FAIL (2025-12-30 10:45:58 UTC) orbiter warning: context deadline exceeded Duration: 5.000467636s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.XX.X.X.X.XX.X.....XX..........XX...X.X..XX.XX........X.....] duration: 11.809569413s trace: https://jaeger.dev.storj.io/trace/5c4f8766c61bbc8a FAIL (2025-12-30 10:45:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: 6OIBW7HMNQR3UXQYWFDP6J2AZDSL35VO4Z6PJ5P7UCRIJFALL77A) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: KVNNUQGW7FKC5F77RAFEFUFSMXZ62FT7RIVO5F6JFZBCGD7YC4TA) context deadline exceeded ======LOGS====== 0 10:41:37.067033916 (595h12m36.938116961s) [upload=521763]: Starting upload Dec 30 10:41:37.067118447 (595h12m36.938201492s) [upload=521763]: Done starting upload Dec 30 10:41:37.067361941 (595h12m36.938444976s) [upload=521763]: Uploading segments... Dec 30 10:41:37.067504051 (595h12m36.938587096s) [upload=521763]: Got next segment. Inline: false Dec 30 10:41:37.067575377 (595h12m36.938658412s) [upload=521763]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:41:37.067583141 (595h12m36.938666186s) [upload=521763]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:41:37.315237828 (595h12m37.186320863s) [seg_pos={0 0}, upload=521763]: Begin upload segment... Dec 30 10:41:37.315281842 (595h12m37.186364877s) [seg_pos={0 0}, upload=521763]: Done begin upload segment. Dec 30 10:41:37.315306599 (595h12m37.186389644s) [noise=true , address=77.90.31.13:4629 , piece_id=SNJ4Z5D4QNHBSNDLSH7W6NROV3X3YGQYB2MNAIKURRUBSTCNIGOA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.315384688 (595h12m37.186467723s) [noise=true , address=145.79.227.20:4439 , piece_id=QKDITA4DSTKAIU2S2JXG5H7JM5FH2JE7CDUHAMLMYVPEGF53VOSQ, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.315849714 (595h12m37.186932759s) [noise=true , address=145.79.227.25:4619 , piece_id=QPAQFNMX4S7OY5YYUXQQKUG7UYHCXBDSKBL6A7S45OQTY6BYWTNA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.316276256 (595h12m37.187359291s) [noise=true , address=77.90.31.16:4735 , piece_id=KVNNUQGW7FKC5F77RAFEFUFSMXZ62FT7RIVO5F6JFZBCGD7YC4TA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.316667430 (595h12m37.187750476s) [noise=true , address=145.79.227.48:4303 , piece_id=CXP7CRYIP4NUZYMTYLRA4N4YLZHSOFOK5B5RF7JXCCAQMTX2GHLA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.317060378 (595h12m37.188143434s) [noise=true , address=145.79.227.19:4077 , piece_id=4CRE76JFUMHITF7OXCAIC455F7L5SJC5ZCDMDSIJJ6AQ4HSK7AMA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.317419994 (595h12m37.188503029s) [noise=true , address=45.91.116.12:4225 , piece_id=7H55Y5PZTFNLY4SNJDXD2PEARFNP6OXRJJGXRDA2UMPM435B4K3A, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.317948861 (595h12m37.189031896s) [noise=true , address=145.79.227.39:4014 , piece_id=Q7CWPW2G3K2O5NZVZQTLZAXJRZG5XYFZZJNP4SHVEIH6RW6CAUOA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318277436 (595h12m37.189360481s) [noise=true , address=145.79.227.17:4970 , piece_id=VFVP5IJDEG4OUH6H6QHDCKNORBC5US4ZG3273FPBFT7BO3DNYNMA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318313034 (595h12m37.189396069s) [noise=true , address=145.79.227.45:4229 , piece_id=ZNC76TIUYUXEMZC5FY22U365CICRNFQCSLIXPVZDBF2KNZ2KDI7Q, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318343201 (595h12m37.189426236s) [noise=true , address=145.79.227.21:4475 , piece_id=WP7NRZXGIFRCI3JVCVCXTVXEFZ7BYZQN5JXLTREWXSKD6ULZYSVQ, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318616161 (595h12m37.189699196s) [noise=true , address=145.79.227.11:4730 , piece_id=QMESJWSNK2KAIDZBQWU2OUNMSPEZFWGQXU3QVNFBVEQXYWHPLVXA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318826802 (595h12m37.189909837s) [noise=true , address=145.79.227.50:4374 , piece_id=G5Y7BV34YTPBC7QK3A5OVO3P7ILJT5WYOOXZBAA6QJXPXVCK5UXQ, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318975465 (595h12m37.19005851s) [noise=true , address=145.79.227.31:4799 , piece_id=JEBR7PFHJBN4UOTVN5HM3NNZ5WX4IWZNBRUFFJ3QSDSTONAFAR7Q, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.319245860 (595h12m37.190328895s) [noise=true , address=77.90.31.11:4554 , piece_id=UUJNARWXIOJM775GZKLOXT4ZHBUYDHVTI5STPK2Z7UHYSTGIOV3A, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.321923038 (595h12m37.193006083s) [noise=true , address=77.90.31.18:4808 , piece_id=6OIBW7HMNQR3UXQYWFDP6J2AZDSL35VO4Z6PJ5P7UCRIJFALL77A, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:38.919914306 (595h12m38.790997351s) [upload=521763]: Next returned nil segment Dec 30 10:41:38.919944133 (595h12m38.791027178s) [upload=521763]: Waiting for error group managing segments... Dec 30 10:41:39.299400313 (595h12m39.170483348s) [noise=true , address=145.79.227.31:4799 , piece_id=JEBR7PFHJBN4UOTVN5HM3NNZ5WX4IWZNBRUFFJ3QSDSTONAFAR7Q, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.301898620 (595h12m39.172981665s) [noise=true , address=145.79.227.21:4475 , piece_id=WP7NRZXGIFRCI3JVCVCXTVXEFZ7BYZQN5JXLTREWXSKD6ULZYSVQ, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.305326327 (595h12m39.176409362s) [noise=true , address=145.79.227.50:4374 , piece_id=G5Y7BV34YTPBC7QK3A5OVO3P7ILJT5WYOOXZBAA6QJXPXVCK5UXQ, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.305621429 (595h12m39.176704474s) [noise=true , address=145.79.227.45:4229 , piece_id=ZNC76TIUYUXEMZC5FY22U365CICRNFQCSLIXPVZDBF2KNZ2KDI7Q, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.313049473 (595h12m39.184132508s) [noise=true , address=145.79.227.48:4303 , piece_id=CXP7CRYIP4NUZYMTYLRA4N4YLZHSOFOK5B5RF7JXCCAQMTX2GHLA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.313096753 (595h12m39.184179798s) [noise=true , address=145.79.227.25:4619 , piece_id=QPAQFNMX4S7OY5YYUXQQKUG7UYHCXBDSKBL6A7S45OQTY6BYWTNA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.330307055 (595h12m39.20139009s) [noise=true , address=145.79.227.17:4970 , piece_id=VFVP5IJDEG4OUH6H6QHDCKNORBC5US4ZG3273FPBFT7BO3DNYNMA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.331102860 (595h12m39.202185895s) [noise=true , address=145.79.227.11:4730 , piece_id=QMESJWSNK2KAIDZBQWU2OUNMSPEZFWGQXU3QVNFBVEQXYWHPLVXA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.333242113 (595h12m39.204325158s) [noise=true , address=145.79.227.19:4077 , piece_id=4CRE76JFUMHITF7OXCAIC455F7L5SJC5ZCDMDSIJJ6AQ4HSK7AMA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.355266472 (595h12m39.226349507s) [noise=true , address=145.79.227.20:4439 , piece_id=QKDITA4DSTKAIU2S2JXG5H7JM5FH2JE7CDUHAMLMYVPEGF53VOSQ, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.387551277 (595h12m39.258634312s) [noise=true , address=145.79.227.39:4014 , piece_id=Q7CWPW2G3K2O5NZVZQTLZAXJRZG5XYFZZJNP4SHVEIH6RW6CAUOA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.876119214 (595h12m39.747202249s) [noise=true , address=77.90.31.16:4735 , piece_id=KVNNUQGW7FKC5F77RAFEFUFSMXZ62FT7RIVO5F6JFZBCGD7YC4TA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:40.191523548 (595h12m40.062606593s) [noise=true , address=77.90.31.18:4808 , piece_id=6OIBW7HMNQR3UXQYWFDP6J2AZDSL35VO4Z6PJ5P7UCRIJFALL77A, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:40.191549366 (595h12m40.062632401s) [seg_pos={0 0}, upload=521763]: Segment reached optimal threshold of 13 pieces. Dec 30 10:41:40.191708269 (595h12m40.062791304s) [noise=true , address=77.90.31.11:4554 , piece_id=UUJNARWXIOJM775GZKLOXT4ZHBUYDHVTI5STPK2Z7UHYSTGIOV3A, seg_pos={0 0}, upload=521763]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:35868->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:35868->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35868->77.90.31.11:4554: use of closed network connection Dec 30 10:41:40.191714180 (595h12m40.062797225s) [noise=true , address=77.90.31.11:4554 , piece_id=UUJNARWXIOJM775GZKLOXT4ZHBUYDHVTI5STPK2Z7UHYSTGIOV3A, seg_pos={0 0}, upload=521763]: Long tail cancellation due to optimal threshold Dec 30 10:41:40.191795394 (595h12m40.062878459s) [noise=true , address=77.90.31.13:4629 , piece_id=SNJ4Z5D4QNHBSNDLSH7W6NROV3X3YGQYB2MNAIKURRUBSTCNIGOA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 30 10:41:40.191814181 (595h12m40.062897236s) [noise=true , address=77.90.31.13:4629 , piece_id=SNJ4Z5D4QNHBSNDLSH7W6NROV3X3YGQYB2MNAIKURRUBSTCNIGOA, seg_pos={0 0}, upload=521763]: Long tail cancellation due to optimal threshold Dec 30 10:41:40.191917547 (595h12m40.063000583s) [noise=true , address=45.91.116.12:4225 , piece_id=7H55Y5PZTFNLY4SNJDXD2PEARFNP6OXRJJGXRDA2UMPM435B4K3A, seg_pos={0 0}, upload=521763]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 30 10:41:40.191923709 (595h12m40.063006744s) [noise=true , address=45.91.116.12:4225 , piece_id=7H55Y5PZTFNLY4SNJDXD2PEARFNP6OXRJJGXRDA2UMPM435B4K3A, seg_pos={0 0}, upload=521763]: Long tail cancellation due to optimal threshold Dec 30 10:41:40.191939890 (595h12m40.063022935s) [seg_pos={0 0}, upload=521763]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:41:40.192001908 (595h12m40.063084943s) [upload=521763]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:41:40.192006397 (595h12m40.063089442s) [upload=521763]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:41:40.486537746 (595h12m40.357620801s) [upload=521763]: Done uploading segments... Dec 30 10:41:40.486564095 (595h12m40.35764715s) [upload=521763]: Upload finished. err: <nil> Duration: 11.809594592s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [.X...X...X........X.....X.X............X......X.X.XXXX.X.XX.] duration: 3.932866611s trace: https://jaeger.dev.storj.io/trace/c7833df3992df96 FAIL (2025-12-30 10:45:11 UTC) orbiter warning: context deadline exceeded Duration: 3.932905323s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X....X.........X....................] duration: 41.5149963s trace: https://jaeger.dev.storj.io/trace/1dc8263b2e55d7dc FAIL (2025-12-30 10:44:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311300688 (594h58m28.182383733s) [noise=true , address=145.79.227.43:4179 , piece_id=CQ32O4X3PKQN37OWC3WU5Z5FF7CCAIOAWLUISOXJTTIXTSMKYNQA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311642950 (594h58m28.182725995s) [noise=true , address=145.79.227.30:4783 , piece_id=UIXZYYXWC7KJZUV3BNZAIUSZNDEYA5RFFLKMXWF3XLNKJF5K2A6Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311712022 (594h58m28.182795057s) [noise=true , address=145.79.227.35:4492 , piece_id=GM7TCP65VSFAGWDCRG4VKWAPJHU5HGZFZTVAD4XM6PJFMLCC4NRA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312038483 (594h58m28.183121528s) [noise=true , address=77.90.31.17:4798 , piece_id=HT7E2VVRR2IURW5OIXQ4BLR55QI2BEKJG3VIRG4CABVPCUZRIGWA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312100291 (594h58m28.183183326s) [noise=true , address=77.90.31.19:4859 , piece_id=HEZMENV33Y6A6MMMMDIUECW44662O24WTIG2RWCWZL3IGNKWCH2A, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312494201 (594h58m28.183577246s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312500252 (594h58m28.183583287s) [noise=true , address=145.79.227.49:4359 , piece_id=RGCUJ32JS6NXVVLH7HWUD5IZNV3HZPRRVIHEBJEII2JPEENVIKZA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312899192 (594h58m28.183982227s) [noise=true , address=145.79.227.18:4037 , piece_id=IQZILX3EVK57WA5QGC4VGRYTYWQX37PLIASYIYXIHWBWKGTSEESA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313265008 (594h58m28.184348053s) [noise=true , address=145.79.227.6:4267 , piece_id=ZE5IVJYY2XEW3QOZBBE7LYICAALYG3DZHU6K7UDYUGBHRTWPQYPA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313640203 (594h58m28.184723248s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313904957 (594h58m28.184988012s) [noise=true , address=45.91.116.14:4213 , piece_id=BUTDQ3PI56Q7ZHNYHCEMWPHJUI42LXBR42ZGF4VKMDXJ47BSPU3Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314038000 (594h58m28.185121045s) [noise=true , address=77.90.31.12:4603 , piece_id=QQ5V6RXCHOHVL2QTWLLRFPJBAQQ5KW47T2N44EBMIUW747Q4ZOAA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314410920 (594h58m28.185493965s) [noise=true , address=145.79.227.44:4219 , piece_id=NSHMZDOI264JBC5ML4GRTH66FKT76C5QQ534UA4QDFEIG5P6CHDQ, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314664292 (594h58m28.185747338s) [noise=true , address=145.79.227.40:4062 , piece_id=RTSWJJIEADDCF5Z4JMT4NERSF7EXCWP7QZKDN4KIWEX54UDLNZEA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.315164305 (594h58m28.18624734s) [noise=true , address=77.90.31.10:4540 , piece_id=5LXEPFE5MNCGS6KJK4RP5Q5MV4M3TQMVR5FK4DYTYY4MKTFXTP2Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:29.778852769 (594h58m29.649935804s) [upload=521557]: Next returned nil segment Dec 30 10:27:29.778904818 (594h58m29.649987863s) [upload=521557]: Waiting for error group managing segments... Dec 30 10:27:30.177976438 (594h58m30.049059474s) [noise=true , address=145.79.227.18:4037 , piece_id=IQZILX3EVK57WA5QGC4VGRYTYWQX37PLIASYIYXIHWBWKGTSEESA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.239401413 (594h58m30.110484448s) [noise=true , address=145.79.227.6:4267 , piece_id=ZE5IVJYY2XEW3QOZBBE7LYICAALYG3DZHU6K7UDYUGBHRTWPQYPA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.247231673 (594h58m30.118314708s) [noise=true , address=145.79.227.35:4492 , piece_id=GM7TCP65VSFAGWDCRG4VKWAPJHU5HGZFZTVAD4XM6PJFMLCC4NRA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.364460186 (594h58m30.235543221s) [noise=true , address=77.90.31.17:4798 , piece_id=HT7E2VVRR2IURW5OIXQ4BLR55QI2BEKJG3VIRG4CABVPCUZRIGWA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.415614833 (594h58m30.286697868s) [noise=true , address=77.90.31.12:4603 , piece_id=QQ5V6RXCHOHVL2QTWLLRFPJBAQQ5KW47T2N44EBMIUW747Q4ZOAA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.444886882 (594h58m30.315969917s) [noise=true , address=77.90.31.10:4540 , piece_id=5LXEPFE5MNCGS6KJK4RP5Q5MV4M3TQMVR5FK4DYTYY4MKTFXTP2Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.512047427 (594h58m30.383130462s) [noise=true , address=45.91.116.14:4213 , piece_id=BUTDQ3PI56Q7ZHNYHCEMWPHJUI42LXBR42ZGF4VKMDXJ47BSPU3Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.536847771 (594h58m30.407930806s) [noise=true , address=77.90.31.19:4859 , piece_id=HEZMENV33Y6A6MMMMDIUECW44662O24WTIG2RWCWZL3IGNKWCH2A, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.680580563 (594h58m30.551663608s) [noise=true , address=45.91.116.12:4253 , piece_id=NPLIO7NLEIRTQAWDQ7WZVZE5ZSRLDPDNM34M6SEI53G5XEHB3LJA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.065501378 (594h58m30.936584414s) [noise=true , address=145.79.227.49:4359 , piece_id=RGCUJ32JS6NXVVLH7HWUD5IZNV3HZPRRVIHEBJEII2JPEENVIKZA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.104555514 (594h58m30.975638559s) [noise=true , address=145.79.227.44:4219 , piece_id=NSHMZDOI264JBC5ML4GRTH66FKT76C5QQ534UA4QDFEIG5P6CHDQ, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.115841635 (594h58m30.98692467s) [noise=true , address=145.79.227.30:4783 , piece_id=UIXZYYXWC7KJZUV3BNZAIUSZNDEYA5RFFLKMXWF3XLNKJF5K2A6Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.364792710 (594h58m31.235875755s) [noise=true , address=145.79.227.40:4062 , piece_id=RTSWJJIEADDCF5Z4JMT4NERSF7EXCWP7QZKDN4KIWEX54UDLNZEA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.374552906 (594h58m31.245635941s) [noise=true , address=145.79.227.43:4179 , piece_id=CQ32O4X3PKQN37OWC3WU5Z5FF7CCAIOAWLUISOXJTTIXTSMKYNQA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.374572984 (594h58m31.245656019s) [seg_pos={0 0}, upload=521557]: Segment reached optimal threshold of 14 pieces. Dec 30 10:27:31.374775900 (594h58m31.245858935s) [noise=true , address=45.91.116.10:4115 , piece_id=DRZ2YOFTPSE7SQD6MJ6YQWRQN6MDOLHHP7LWJMKMCPHO37A72C4A, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1arcuTW5rwnvfiqi2H7dL3oL6jKBkDeGq6DmgRGpS9Vs9BCHmf): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.374782553 (594h58m31.245865598s) [noise=true , address=45.91.116.10:4115 , piece_id=DRZ2YOFTPSE7SQD6MJ6YQWRQN6MDOLHHP7LWJMKMCPHO37A72C4A, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.374972405 (594h58m31.24605544s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.374978286 (594h58m31.246061321s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375019294 (594h58m31.246102349s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.375038921 (594h58m31.246121966s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375091411 (594h58m31.246174446s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE): protocol: write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection; write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection Dec 30 10:27:31.375099606 (594h58m31.246182641s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375195900 (594h58m31.246278935s) [seg_pos={0 0}, upload=521557]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:27:31.375261705 (594h58m31.24634474s) [upload=521557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:27:31.375266604 (594h58m31.246349639s) [upload=521557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:27:31.664767884 (594h58m31.53585099s) [upload=521557]: Done uploading segments... Dec 30 10:27:31.664793884 (594h58m31.535876929s) [upload=521557]: Upload finished. err: <nil> Duration: 41.515016117s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.XX.X.X.X.XX.X.....XX..........XX...X.X..XX.XX........X....] duration: 5.866975764s trace: https://jaeger.dev.storj.io/trace/5c4f8766c61bbc8a FAIL (2025-12-30 10:44:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: 6OIBW7HMNQR3UXQYWFDP6J2AZDSL35VO4Z6PJ5P7UCRIJFALL77A) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: KVNNUQGW7FKC5F77RAFEFUFSMXZ62FT7RIVO5F6JFZBCGD7YC4TA) context deadline exceeded ======LOGS====== 0 10:41:37.067033916 (595h12m36.938116961s) [upload=521763]: Starting upload Dec 30 10:41:37.067118447 (595h12m36.938201492s) [upload=521763]: Done starting upload Dec 30 10:41:37.067361941 (595h12m36.938444976s) [upload=521763]: Uploading segments... Dec 30 10:41:37.067504051 (595h12m36.938587096s) [upload=521763]: Got next segment. Inline: false Dec 30 10:41:37.067575377 (595h12m36.938658412s) [upload=521763]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:41:37.067583141 (595h12m36.938666186s) [upload=521763]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:41:37.315237828 (595h12m37.186320863s) [seg_pos={0 0}, upload=521763]: Begin upload segment... Dec 30 10:41:37.315281842 (595h12m37.186364877s) [seg_pos={0 0}, upload=521763]: Done begin upload segment. Dec 30 10:41:37.315306599 (595h12m37.186389644s) [noise=true , address=77.90.31.13:4629 , piece_id=SNJ4Z5D4QNHBSNDLSH7W6NROV3X3YGQYB2MNAIKURRUBSTCNIGOA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.315384688 (595h12m37.186467723s) [noise=true , address=145.79.227.20:4439 , piece_id=QKDITA4DSTKAIU2S2JXG5H7JM5FH2JE7CDUHAMLMYVPEGF53VOSQ, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.315849714 (595h12m37.186932759s) [noise=true , address=145.79.227.25:4619 , piece_id=QPAQFNMX4S7OY5YYUXQQKUG7UYHCXBDSKBL6A7S45OQTY6BYWTNA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.316276256 (595h12m37.187359291s) [noise=true , address=77.90.31.16:4735 , piece_id=KVNNUQGW7FKC5F77RAFEFUFSMXZ62FT7RIVO5F6JFZBCGD7YC4TA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.316667430 (595h12m37.187750476s) [noise=true , address=145.79.227.48:4303 , piece_id=CXP7CRYIP4NUZYMTYLRA4N4YLZHSOFOK5B5RF7JXCCAQMTX2GHLA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.317060378 (595h12m37.188143434s) [noise=true , address=145.79.227.19:4077 , piece_id=4CRE76JFUMHITF7OXCAIC455F7L5SJC5ZCDMDSIJJ6AQ4HSK7AMA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.317419994 (595h12m37.188503029s) [noise=true , address=45.91.116.12:4225 , piece_id=7H55Y5PZTFNLY4SNJDXD2PEARFNP6OXRJJGXRDA2UMPM435B4K3A, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.317948861 (595h12m37.189031896s) [noise=true , address=145.79.227.39:4014 , piece_id=Q7CWPW2G3K2O5NZVZQTLZAXJRZG5XYFZZJNP4SHVEIH6RW6CAUOA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318277436 (595h12m37.189360481s) [noise=true , address=145.79.227.17:4970 , piece_id=VFVP5IJDEG4OUH6H6QHDCKNORBC5US4ZG3273FPBFT7BO3DNYNMA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318313034 (595h12m37.189396069s) [noise=true , address=145.79.227.45:4229 , piece_id=ZNC76TIUYUXEMZC5FY22U365CICRNFQCSLIXPVZDBF2KNZ2KDI7Q, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318343201 (595h12m37.189426236s) [noise=true , address=145.79.227.21:4475 , piece_id=WP7NRZXGIFRCI3JVCVCXTVXEFZ7BYZQN5JXLTREWXSKD6ULZYSVQ, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318616161 (595h12m37.189699196s) [noise=true , address=145.79.227.11:4730 , piece_id=QMESJWSNK2KAIDZBQWU2OUNMSPEZFWGQXU3QVNFBVEQXYWHPLVXA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318826802 (595h12m37.189909837s) [noise=true , address=145.79.227.50:4374 , piece_id=G5Y7BV34YTPBC7QK3A5OVO3P7ILJT5WYOOXZBAA6QJXPXVCK5UXQ, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318975465 (595h12m37.19005851s) [noise=true , address=145.79.227.31:4799 , piece_id=JEBR7PFHJBN4UOTVN5HM3NNZ5WX4IWZNBRUFFJ3QSDSTONAFAR7Q, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.319245860 (595h12m37.190328895s) [noise=true , address=77.90.31.11:4554 , piece_id=UUJNARWXIOJM775GZKLOXT4ZHBUYDHVTI5STPK2Z7UHYSTGIOV3A, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.321923038 (595h12m37.193006083s) [noise=true , address=77.90.31.18:4808 , piece_id=6OIBW7HMNQR3UXQYWFDP6J2AZDSL35VO4Z6PJ5P7UCRIJFALL77A, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:38.919914306 (595h12m38.790997351s) [upload=521763]: Next returned nil segment Dec 30 10:41:38.919944133 (595h12m38.791027178s) [upload=521763]: Waiting for error group managing segments... Dec 30 10:41:39.299400313 (595h12m39.170483348s) [noise=true , address=145.79.227.31:4799 , piece_id=JEBR7PFHJBN4UOTVN5HM3NNZ5WX4IWZNBRUFFJ3QSDSTONAFAR7Q, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.301898620 (595h12m39.172981665s) [noise=true , address=145.79.227.21:4475 , piece_id=WP7NRZXGIFRCI3JVCVCXTVXEFZ7BYZQN5JXLTREWXSKD6ULZYSVQ, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.305326327 (595h12m39.176409362s) [noise=true , address=145.79.227.50:4374 , piece_id=G5Y7BV34YTPBC7QK3A5OVO3P7ILJT5WYOOXZBAA6QJXPXVCK5UXQ, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.305621429 (595h12m39.176704474s) [noise=true , address=145.79.227.45:4229 , piece_id=ZNC76TIUYUXEMZC5FY22U365CICRNFQCSLIXPVZDBF2KNZ2KDI7Q, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.313049473 (595h12m39.184132508s) [noise=true , address=145.79.227.48:4303 , piece_id=CXP7CRYIP4NUZYMTYLRA4N4YLZHSOFOK5B5RF7JXCCAQMTX2GHLA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.313096753 (595h12m39.184179798s) [noise=true , address=145.79.227.25:4619 , piece_id=QPAQFNMX4S7OY5YYUXQQKUG7UYHCXBDSKBL6A7S45OQTY6BYWTNA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.330307055 (595h12m39.20139009s) [noise=true , address=145.79.227.17:4970 , piece_id=VFVP5IJDEG4OUH6H6QHDCKNORBC5US4ZG3273FPBFT7BO3DNYNMA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.331102860 (595h12m39.202185895s) [noise=true , address=145.79.227.11:4730 , piece_id=QMESJWSNK2KAIDZBQWU2OUNMSPEZFWGQXU3QVNFBVEQXYWHPLVXA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.333242113 (595h12m39.204325158s) [noise=true , address=145.79.227.19:4077 , piece_id=4CRE76JFUMHITF7OXCAIC455F7L5SJC5ZCDMDSIJJ6AQ4HSK7AMA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.355266472 (595h12m39.226349507s) [noise=true , address=145.79.227.20:4439 , piece_id=QKDITA4DSTKAIU2S2JXG5H7JM5FH2JE7CDUHAMLMYVPEGF53VOSQ, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.387551277 (595h12m39.258634312s) [noise=true , address=145.79.227.39:4014 , piece_id=Q7CWPW2G3K2O5NZVZQTLZAXJRZG5XYFZZJNP4SHVEIH6RW6CAUOA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.876119214 (595h12m39.747202249s) [noise=true , address=77.90.31.16:4735 , piece_id=KVNNUQGW7FKC5F77RAFEFUFSMXZ62FT7RIVO5F6JFZBCGD7YC4TA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:40.191523548 (595h12m40.062606593s) [noise=true , address=77.90.31.18:4808 , piece_id=6OIBW7HMNQR3UXQYWFDP6J2AZDSL35VO4Z6PJ5P7UCRIJFALL77A, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:40.191549366 (595h12m40.062632401s) [seg_pos={0 0}, upload=521763]: Segment reached optimal threshold of 13 pieces. Dec 30 10:41:40.191708269 (595h12m40.062791304s) [noise=true , address=77.90.31.11:4554 , piece_id=UUJNARWXIOJM775GZKLOXT4ZHBUYDHVTI5STPK2Z7UHYSTGIOV3A, seg_pos={0 0}, upload=521763]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:35868->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:35868->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35868->77.90.31.11:4554: use of closed network connection Dec 30 10:41:40.191714180 (595h12m40.062797225s) [noise=true , address=77.90.31.11:4554 , piece_id=UUJNARWXIOJM775GZKLOXT4ZHBUYDHVTI5STPK2Z7UHYSTGIOV3A, seg_pos={0 0}, upload=521763]: Long tail cancellation due to optimal threshold Dec 30 10:41:40.191795394 (595h12m40.062878459s) [noise=true , address=77.90.31.13:4629 , piece_id=SNJ4Z5D4QNHBSNDLSH7W6NROV3X3YGQYB2MNAIKURRUBSTCNIGOA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 30 10:41:40.191814181 (595h12m40.062897236s) [noise=true , address=77.90.31.13:4629 , piece_id=SNJ4Z5D4QNHBSNDLSH7W6NROV3X3YGQYB2MNAIKURRUBSTCNIGOA, seg_pos={0 0}, upload=521763]: Long tail cancellation due to optimal threshold Dec 30 10:41:40.191917547 (595h12m40.063000583s) [noise=true , address=45.91.116.12:4225 , piece_id=7H55Y5PZTFNLY4SNJDXD2PEARFNP6OXRJJGXRDA2UMPM435B4K3A, seg_pos={0 0}, upload=521763]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 30 10:41:40.191923709 (595h12m40.063006744s) [noise=true , address=45.91.116.12:4225 , piece_id=7H55Y5PZTFNLY4SNJDXD2PEARFNP6OXRJJGXRDA2UMPM435B4K3A, seg_pos={0 0}, upload=521763]: Long tail cancellation due to optimal threshold Dec 30 10:41:40.191939890 (595h12m40.063022935s) [seg_pos={0 0}, upload=521763]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:41:40.192001908 (595h12m40.063084943s) [upload=521763]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:41:40.192006397 (595h12m40.063089442s) [upload=521763]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:41:40.486537746 (595h12m40.357620801s) [upload=521763]: Done uploading segments... Dec 30 10:41:40.486564095 (595h12m40.35764715s) [upload=521763]: Upload finished. err: <nil> Duration: 5.867009079s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [..X...X...X........X.....X.X............X......X.X.XXXX.X.XX] duration: 5.001597757s trace: https://jaeger.dev.storj.io/trace/c7833df3992df96 FAIL (2025-12-30 10:44:08 UTC) orbiter warning: context deadline exceeded Duration: 5.001624038s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X....X.........X...................] duration: 29.584993494s trace: https://jaeger.dev.storj.io/trace/1dc8263b2e55d7dc FAIL (2025-12-30 10:43:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311300688 (594h58m28.182383733s) [noise=true , address=145.79.227.43:4179 , piece_id=CQ32O4X3PKQN37OWC3WU5Z5FF7CCAIOAWLUISOXJTTIXTSMKYNQA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311642950 (594h58m28.182725995s) [noise=true , address=145.79.227.30:4783 , piece_id=UIXZYYXWC7KJZUV3BNZAIUSZNDEYA5RFFLKMXWF3XLNKJF5K2A6Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311712022 (594h58m28.182795057s) [noise=true , address=145.79.227.35:4492 , piece_id=GM7TCP65VSFAGWDCRG4VKWAPJHU5HGZFZTVAD4XM6PJFMLCC4NRA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312038483 (594h58m28.183121528s) [noise=true , address=77.90.31.17:4798 , piece_id=HT7E2VVRR2IURW5OIXQ4BLR55QI2BEKJG3VIRG4CABVPCUZRIGWA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312100291 (594h58m28.183183326s) [noise=true , address=77.90.31.19:4859 , piece_id=HEZMENV33Y6A6MMMMDIUECW44662O24WTIG2RWCWZL3IGNKWCH2A, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312494201 (594h58m28.183577246s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312500252 (594h58m28.183583287s) [noise=true , address=145.79.227.49:4359 , piece_id=RGCUJ32JS6NXVVLH7HWUD5IZNV3HZPRRVIHEBJEII2JPEENVIKZA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312899192 (594h58m28.183982227s) [noise=true , address=145.79.227.18:4037 , piece_id=IQZILX3EVK57WA5QGC4VGRYTYWQX37PLIASYIYXIHWBWKGTSEESA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313265008 (594h58m28.184348053s) [noise=true , address=145.79.227.6:4267 , piece_id=ZE5IVJYY2XEW3QOZBBE7LYICAALYG3DZHU6K7UDYUGBHRTWPQYPA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313640203 (594h58m28.184723248s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313904957 (594h58m28.184988012s) [noise=true , address=45.91.116.14:4213 , piece_id=BUTDQ3PI56Q7ZHNYHCEMWPHJUI42LXBR42ZGF4VKMDXJ47BSPU3Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314038000 (594h58m28.185121045s) [noise=true , address=77.90.31.12:4603 , piece_id=QQ5V6RXCHOHVL2QTWLLRFPJBAQQ5KW47T2N44EBMIUW747Q4ZOAA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314410920 (594h58m28.185493965s) [noise=true , address=145.79.227.44:4219 , piece_id=NSHMZDOI264JBC5ML4GRTH66FKT76C5QQ534UA4QDFEIG5P6CHDQ, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314664292 (594h58m28.185747338s) [noise=true , address=145.79.227.40:4062 , piece_id=RTSWJJIEADDCF5Z4JMT4NERSF7EXCWP7QZKDN4KIWEX54UDLNZEA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.315164305 (594h58m28.18624734s) [noise=true , address=77.90.31.10:4540 , piece_id=5LXEPFE5MNCGS6KJK4RP5Q5MV4M3TQMVR5FK4DYTYY4MKTFXTP2Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:29.778852769 (594h58m29.649935804s) [upload=521557]: Next returned nil segment Dec 30 10:27:29.778904818 (594h58m29.649987863s) [upload=521557]: Waiting for error group managing segments... Dec 30 10:27:30.177976438 (594h58m30.049059474s) [noise=true , address=145.79.227.18:4037 , piece_id=IQZILX3EVK57WA5QGC4VGRYTYWQX37PLIASYIYXIHWBWKGTSEESA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.239401413 (594h58m30.110484448s) [noise=true , address=145.79.227.6:4267 , piece_id=ZE5IVJYY2XEW3QOZBBE7LYICAALYG3DZHU6K7UDYUGBHRTWPQYPA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.247231673 (594h58m30.118314708s) [noise=true , address=145.79.227.35:4492 , piece_id=GM7TCP65VSFAGWDCRG4VKWAPJHU5HGZFZTVAD4XM6PJFMLCC4NRA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.364460186 (594h58m30.235543221s) [noise=true , address=77.90.31.17:4798 , piece_id=HT7E2VVRR2IURW5OIXQ4BLR55QI2BEKJG3VIRG4CABVPCUZRIGWA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.415614833 (594h58m30.286697868s) [noise=true , address=77.90.31.12:4603 , piece_id=QQ5V6RXCHOHVL2QTWLLRFPJBAQQ5KW47T2N44EBMIUW747Q4ZOAA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.444886882 (594h58m30.315969917s) [noise=true , address=77.90.31.10:4540 , piece_id=5LXEPFE5MNCGS6KJK4RP5Q5MV4M3TQMVR5FK4DYTYY4MKTFXTP2Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.512047427 (594h58m30.383130462s) [noise=true , address=45.91.116.14:4213 , piece_id=BUTDQ3PI56Q7ZHNYHCEMWPHJUI42LXBR42ZGF4VKMDXJ47BSPU3Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.536847771 (594h58m30.407930806s) [noise=true , address=77.90.31.19:4859 , piece_id=HEZMENV33Y6A6MMMMDIUECW44662O24WTIG2RWCWZL3IGNKWCH2A, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.680580563 (594h58m30.551663608s) [noise=true , address=45.91.116.12:4253 , piece_id=NPLIO7NLEIRTQAWDQ7WZVZE5ZSRLDPDNM34M6SEI53G5XEHB3LJA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.065501378 (594h58m30.936584414s) [noise=true , address=145.79.227.49:4359 , piece_id=RGCUJ32JS6NXVVLH7HWUD5IZNV3HZPRRVIHEBJEII2JPEENVIKZA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.104555514 (594h58m30.975638559s) [noise=true , address=145.79.227.44:4219 , piece_id=NSHMZDOI264JBC5ML4GRTH66FKT76C5QQ534UA4QDFEIG5P6CHDQ, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.115841635 (594h58m30.98692467s) [noise=true , address=145.79.227.30:4783 , piece_id=UIXZYYXWC7KJZUV3BNZAIUSZNDEYA5RFFLKMXWF3XLNKJF5K2A6Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.364792710 (594h58m31.235875755s) [noise=true , address=145.79.227.40:4062 , piece_id=RTSWJJIEADDCF5Z4JMT4NERSF7EXCWP7QZKDN4KIWEX54UDLNZEA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.374552906 (594h58m31.245635941s) [noise=true , address=145.79.227.43:4179 , piece_id=CQ32O4X3PKQN37OWC3WU5Z5FF7CCAIOAWLUISOXJTTIXTSMKYNQA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.374572984 (594h58m31.245656019s) [seg_pos={0 0}, upload=521557]: Segment reached optimal threshold of 14 pieces. Dec 30 10:27:31.374775900 (594h58m31.245858935s) [noise=true , address=45.91.116.10:4115 , piece_id=DRZ2YOFTPSE7SQD6MJ6YQWRQN6MDOLHHP7LWJMKMCPHO37A72C4A, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1arcuTW5rwnvfiqi2H7dL3oL6jKBkDeGq6DmgRGpS9Vs9BCHmf): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.374782553 (594h58m31.245865598s) [noise=true , address=45.91.116.10:4115 , piece_id=DRZ2YOFTPSE7SQD6MJ6YQWRQN6MDOLHHP7LWJMKMCPHO37A72C4A, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.374972405 (594h58m31.24605544s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.374978286 (594h58m31.246061321s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375019294 (594h58m31.246102349s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.375038921 (594h58m31.246121966s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375091411 (594h58m31.246174446s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE): protocol: write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection; write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection Dec 30 10:27:31.375099606 (594h58m31.246182641s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375195900 (594h58m31.246278935s) [seg_pos={0 0}, upload=521557]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:27:31.375261705 (594h58m31.24634474s) [upload=521557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:27:31.375266604 (594h58m31.246349639s) [upload=521557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:27:31.664767884 (594h58m31.53585099s) [upload=521557]: Done uploading segments... Dec 30 10:27:31.664793884 (594h58m31.535876929s) [upload=521557]: Upload finished. err: <nil> Duration: 29.585034333s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [...X...X...X........X.....X.X............X......X.X.XXXX.X.X] duration: 5.001675083s trace: https://jaeger.dev.storj.io/trace/58bf975a4e633f7 FAIL (2025-12-30 10:43:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001711815s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X.XX.X.X.X.XX.X.....XX..........XX...X.X..XX.XX........X...] duration: 5.773876918s trace: https://jaeger.dev.storj.io/trace/5c4f8766c61bbc8a FAIL (2025-12-30 10:43:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: 6OIBW7HMNQR3UXQYWFDP6J2AZDSL35VO4Z6PJ5P7UCRIJFALL77A) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: KVNNUQGW7FKC5F77RAFEFUFSMXZ62FT7RIVO5F6JFZBCGD7YC4TA) context deadline exceeded ======LOGS====== 0 10:41:37.067033916 (595h12m36.938116961s) [upload=521763]: Starting upload Dec 30 10:41:37.067118447 (595h12m36.938201492s) [upload=521763]: Done starting upload Dec 30 10:41:37.067361941 (595h12m36.938444976s) [upload=521763]: Uploading segments... Dec 30 10:41:37.067504051 (595h12m36.938587096s) [upload=521763]: Got next segment. Inline: false Dec 30 10:41:37.067575377 (595h12m36.938658412s) [upload=521763]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:41:37.067583141 (595h12m36.938666186s) [upload=521763]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:41:37.315237828 (595h12m37.186320863s) [seg_pos={0 0}, upload=521763]: Begin upload segment... Dec 30 10:41:37.315281842 (595h12m37.186364877s) [seg_pos={0 0}, upload=521763]: Done begin upload segment. Dec 30 10:41:37.315306599 (595h12m37.186389644s) [noise=true , address=77.90.31.13:4629 , piece_id=SNJ4Z5D4QNHBSNDLSH7W6NROV3X3YGQYB2MNAIKURRUBSTCNIGOA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.315384688 (595h12m37.186467723s) [noise=true , address=145.79.227.20:4439 , piece_id=QKDITA4DSTKAIU2S2JXG5H7JM5FH2JE7CDUHAMLMYVPEGF53VOSQ, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.315849714 (595h12m37.186932759s) [noise=true , address=145.79.227.25:4619 , piece_id=QPAQFNMX4S7OY5YYUXQQKUG7UYHCXBDSKBL6A7S45OQTY6BYWTNA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.316276256 (595h12m37.187359291s) [noise=true , address=77.90.31.16:4735 , piece_id=KVNNUQGW7FKC5F77RAFEFUFSMXZ62FT7RIVO5F6JFZBCGD7YC4TA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.316667430 (595h12m37.187750476s) [noise=true , address=145.79.227.48:4303 , piece_id=CXP7CRYIP4NUZYMTYLRA4N4YLZHSOFOK5B5RF7JXCCAQMTX2GHLA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.317060378 (595h12m37.188143434s) [noise=true , address=145.79.227.19:4077 , piece_id=4CRE76JFUMHITF7OXCAIC455F7L5SJC5ZCDMDSIJJ6AQ4HSK7AMA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.317419994 (595h12m37.188503029s) [noise=true , address=45.91.116.12:4225 , piece_id=7H55Y5PZTFNLY4SNJDXD2PEARFNP6OXRJJGXRDA2UMPM435B4K3A, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.317948861 (595h12m37.189031896s) [noise=true , address=145.79.227.39:4014 , piece_id=Q7CWPW2G3K2O5NZVZQTLZAXJRZG5XYFZZJNP4SHVEIH6RW6CAUOA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318277436 (595h12m37.189360481s) [noise=true , address=145.79.227.17:4970 , piece_id=VFVP5IJDEG4OUH6H6QHDCKNORBC5US4ZG3273FPBFT7BO3DNYNMA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318313034 (595h12m37.189396069s) [noise=true , address=145.79.227.45:4229 , piece_id=ZNC76TIUYUXEMZC5FY22U365CICRNFQCSLIXPVZDBF2KNZ2KDI7Q, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318343201 (595h12m37.189426236s) [noise=true , address=145.79.227.21:4475 , piece_id=WP7NRZXGIFRCI3JVCVCXTVXEFZ7BYZQN5JXLTREWXSKD6ULZYSVQ, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318616161 (595h12m37.189699196s) [noise=true , address=145.79.227.11:4730 , piece_id=QMESJWSNK2KAIDZBQWU2OUNMSPEZFWGQXU3QVNFBVEQXYWHPLVXA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318826802 (595h12m37.189909837s) [noise=true , address=145.79.227.50:4374 , piece_id=G5Y7BV34YTPBC7QK3A5OVO3P7ILJT5WYOOXZBAA6QJXPXVCK5UXQ, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318975465 (595h12m37.19005851s) [noise=true , address=145.79.227.31:4799 , piece_id=JEBR7PFHJBN4UOTVN5HM3NNZ5WX4IWZNBRUFFJ3QSDSTONAFAR7Q, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.319245860 (595h12m37.190328895s) [noise=true , address=77.90.31.11:4554 , piece_id=UUJNARWXIOJM775GZKLOXT4ZHBUYDHVTI5STPK2Z7UHYSTGIOV3A, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.321923038 (595h12m37.193006083s) [noise=true , address=77.90.31.18:4808 , piece_id=6OIBW7HMNQR3UXQYWFDP6J2AZDSL35VO4Z6PJ5P7UCRIJFALL77A, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:38.919914306 (595h12m38.790997351s) [upload=521763]: Next returned nil segment Dec 30 10:41:38.919944133 (595h12m38.791027178s) [upload=521763]: Waiting for error group managing segments... Dec 30 10:41:39.299400313 (595h12m39.170483348s) [noise=true , address=145.79.227.31:4799 , piece_id=JEBR7PFHJBN4UOTVN5HM3NNZ5WX4IWZNBRUFFJ3QSDSTONAFAR7Q, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.301898620 (595h12m39.172981665s) [noise=true , address=145.79.227.21:4475 , piece_id=WP7NRZXGIFRCI3JVCVCXTVXEFZ7BYZQN5JXLTREWXSKD6ULZYSVQ, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.305326327 (595h12m39.176409362s) [noise=true , address=145.79.227.50:4374 , piece_id=G5Y7BV34YTPBC7QK3A5OVO3P7ILJT5WYOOXZBAA6QJXPXVCK5UXQ, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.305621429 (595h12m39.176704474s) [noise=true , address=145.79.227.45:4229 , piece_id=ZNC76TIUYUXEMZC5FY22U365CICRNFQCSLIXPVZDBF2KNZ2KDI7Q, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.313049473 (595h12m39.184132508s) [noise=true , address=145.79.227.48:4303 , piece_id=CXP7CRYIP4NUZYMTYLRA4N4YLZHSOFOK5B5RF7JXCCAQMTX2GHLA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.313096753 (595h12m39.184179798s) [noise=true , address=145.79.227.25:4619 , piece_id=QPAQFNMX4S7OY5YYUXQQKUG7UYHCXBDSKBL6A7S45OQTY6BYWTNA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.330307055 (595h12m39.20139009s) [noise=true , address=145.79.227.17:4970 , piece_id=VFVP5IJDEG4OUH6H6QHDCKNORBC5US4ZG3273FPBFT7BO3DNYNMA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.331102860 (595h12m39.202185895s) [noise=true , address=145.79.227.11:4730 , piece_id=QMESJWSNK2KAIDZBQWU2OUNMSPEZFWGQXU3QVNFBVEQXYWHPLVXA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.333242113 (595h12m39.204325158s) [noise=true , address=145.79.227.19:4077 , piece_id=4CRE76JFUMHITF7OXCAIC455F7L5SJC5ZCDMDSIJJ6AQ4HSK7AMA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.355266472 (595h12m39.226349507s) [noise=true , address=145.79.227.20:4439 , piece_id=QKDITA4DSTKAIU2S2JXG5H7JM5FH2JE7CDUHAMLMYVPEGF53VOSQ, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.387551277 (595h12m39.258634312s) [noise=true , address=145.79.227.39:4014 , piece_id=Q7CWPW2G3K2O5NZVZQTLZAXJRZG5XYFZZJNP4SHVEIH6RW6CAUOA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.876119214 (595h12m39.747202249s) [noise=true , address=77.90.31.16:4735 , piece_id=KVNNUQGW7FKC5F77RAFEFUFSMXZ62FT7RIVO5F6JFZBCGD7YC4TA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:40.191523548 (595h12m40.062606593s) [noise=true , address=77.90.31.18:4808 , piece_id=6OIBW7HMNQR3UXQYWFDP6J2AZDSL35VO4Z6PJ5P7UCRIJFALL77A, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:40.191549366 (595h12m40.062632401s) [seg_pos={0 0}, upload=521763]: Segment reached optimal threshold of 13 pieces. Dec 30 10:41:40.191708269 (595h12m40.062791304s) [noise=true , address=77.90.31.11:4554 , piece_id=UUJNARWXIOJM775GZKLOXT4ZHBUYDHVTI5STPK2Z7UHYSTGIOV3A, seg_pos={0 0}, upload=521763]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:35868->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:35868->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35868->77.90.31.11:4554: use of closed network connection Dec 30 10:41:40.191714180 (595h12m40.062797225s) [noise=true , address=77.90.31.11:4554 , piece_id=UUJNARWXIOJM775GZKLOXT4ZHBUYDHVTI5STPK2Z7UHYSTGIOV3A, seg_pos={0 0}, upload=521763]: Long tail cancellation due to optimal threshold Dec 30 10:41:40.191795394 (595h12m40.062878459s) [noise=true , address=77.90.31.13:4629 , piece_id=SNJ4Z5D4QNHBSNDLSH7W6NROV3X3YGQYB2MNAIKURRUBSTCNIGOA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 30 10:41:40.191814181 (595h12m40.062897236s) [noise=true , address=77.90.31.13:4629 , piece_id=SNJ4Z5D4QNHBSNDLSH7W6NROV3X3YGQYB2MNAIKURRUBSTCNIGOA, seg_pos={0 0}, upload=521763]: Long tail cancellation due to optimal threshold Dec 30 10:41:40.191917547 (595h12m40.063000583s) [noise=true , address=45.91.116.12:4225 , piece_id=7H55Y5PZTFNLY4SNJDXD2PEARFNP6OXRJJGXRDA2UMPM435B4K3A, seg_pos={0 0}, upload=521763]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 30 10:41:40.191923709 (595h12m40.063006744s) [noise=true , address=45.91.116.12:4225 , piece_id=7H55Y5PZTFNLY4SNJDXD2PEARFNP6OXRJJGXRDA2UMPM435B4K3A, seg_pos={0 0}, upload=521763]: Long tail cancellation due to optimal threshold Dec 30 10:41:40.191939890 (595h12m40.063022935s) [seg_pos={0 0}, upload=521763]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:41:40.192001908 (595h12m40.063084943s) [upload=521763]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:41:40.192006397 (595h12m40.063089442s) [upload=521763]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:41:40.486537746 (595h12m40.357620801s) [upload=521763]: Done uploading segments... Dec 30 10:41:40.486564095 (595h12m40.35764715s) [upload=521763]: Upload finished. err: <nil> Duration: 5.773903099s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.X.XX.X.X.X.XX.X.....XX..........XX...X.X..XX.XX........X..] duration: 5.746356836s trace: https://jaeger.dev.storj.io/trace/5c4f8766c61bbc8a FAIL (2025-12-30 10:43:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: 6OIBW7HMNQR3UXQYWFDP6J2AZDSL35VO4Z6PJ5P7UCRIJFALL77A) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: KVNNUQGW7FKC5F77RAFEFUFSMXZ62FT7RIVO5F6JFZBCGD7YC4TA) context deadline exceeded ======LOGS====== 0 10:41:37.067033916 (595h12m36.938116961s) [upload=521763]: Starting upload Dec 30 10:41:37.067118447 (595h12m36.938201492s) [upload=521763]: Done starting upload Dec 30 10:41:37.067361941 (595h12m36.938444976s) [upload=521763]: Uploading segments... Dec 30 10:41:37.067504051 (595h12m36.938587096s) [upload=521763]: Got next segment. Inline: false Dec 30 10:41:37.067575377 (595h12m36.938658412s) [upload=521763]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:41:37.067583141 (595h12m36.938666186s) [upload=521763]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:41:37.315237828 (595h12m37.186320863s) [seg_pos={0 0}, upload=521763]: Begin upload segment... Dec 30 10:41:37.315281842 (595h12m37.186364877s) [seg_pos={0 0}, upload=521763]: Done begin upload segment. Dec 30 10:41:37.315306599 (595h12m37.186389644s) [noise=true , address=77.90.31.13:4629 , piece_id=SNJ4Z5D4QNHBSNDLSH7W6NROV3X3YGQYB2MNAIKURRUBSTCNIGOA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.315384688 (595h12m37.186467723s) [noise=true , address=145.79.227.20:4439 , piece_id=QKDITA4DSTKAIU2S2JXG5H7JM5FH2JE7CDUHAMLMYVPEGF53VOSQ, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.315849714 (595h12m37.186932759s) [noise=true , address=145.79.227.25:4619 , piece_id=QPAQFNMX4S7OY5YYUXQQKUG7UYHCXBDSKBL6A7S45OQTY6BYWTNA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.316276256 (595h12m37.187359291s) [noise=true , address=77.90.31.16:4735 , piece_id=KVNNUQGW7FKC5F77RAFEFUFSMXZ62FT7RIVO5F6JFZBCGD7YC4TA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.316667430 (595h12m37.187750476s) [noise=true , address=145.79.227.48:4303 , piece_id=CXP7CRYIP4NUZYMTYLRA4N4YLZHSOFOK5B5RF7JXCCAQMTX2GHLA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.317060378 (595h12m37.188143434s) [noise=true , address=145.79.227.19:4077 , piece_id=4CRE76JFUMHITF7OXCAIC455F7L5SJC5ZCDMDSIJJ6AQ4HSK7AMA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.317419994 (595h12m37.188503029s) [noise=true , address=45.91.116.12:4225 , piece_id=7H55Y5PZTFNLY4SNJDXD2PEARFNP6OXRJJGXRDA2UMPM435B4K3A, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.317948861 (595h12m37.189031896s) [noise=true , address=145.79.227.39:4014 , piece_id=Q7CWPW2G3K2O5NZVZQTLZAXJRZG5XYFZZJNP4SHVEIH6RW6CAUOA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318277436 (595h12m37.189360481s) [noise=true , address=145.79.227.17:4970 , piece_id=VFVP5IJDEG4OUH6H6QHDCKNORBC5US4ZG3273FPBFT7BO3DNYNMA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318313034 (595h12m37.189396069s) [noise=true , address=145.79.227.45:4229 , piece_id=ZNC76TIUYUXEMZC5FY22U365CICRNFQCSLIXPVZDBF2KNZ2KDI7Q, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318343201 (595h12m37.189426236s) [noise=true , address=145.79.227.21:4475 , piece_id=WP7NRZXGIFRCI3JVCVCXTVXEFZ7BYZQN5JXLTREWXSKD6ULZYSVQ, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318616161 (595h12m37.189699196s) [noise=true , address=145.79.227.11:4730 , piece_id=QMESJWSNK2KAIDZBQWU2OUNMSPEZFWGQXU3QVNFBVEQXYWHPLVXA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318826802 (595h12m37.189909837s) [noise=true , address=145.79.227.50:4374 , piece_id=G5Y7BV34YTPBC7QK3A5OVO3P7ILJT5WYOOXZBAA6QJXPXVCK5UXQ, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318975465 (595h12m37.19005851s) [noise=true , address=145.79.227.31:4799 , piece_id=JEBR7PFHJBN4UOTVN5HM3NNZ5WX4IWZNBRUFFJ3QSDSTONAFAR7Q, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.319245860 (595h12m37.190328895s) [noise=true , address=77.90.31.11:4554 , piece_id=UUJNARWXIOJM775GZKLOXT4ZHBUYDHVTI5STPK2Z7UHYSTGIOV3A, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.321923038 (595h12m37.193006083s) [noise=true , address=77.90.31.18:4808 , piece_id=6OIBW7HMNQR3UXQYWFDP6J2AZDSL35VO4Z6PJ5P7UCRIJFALL77A, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:38.919914306 (595h12m38.790997351s) [upload=521763]: Next returned nil segment Dec 30 10:41:38.919944133 (595h12m38.791027178s) [upload=521763]: Waiting for error group managing segments... Dec 30 10:41:39.299400313 (595h12m39.170483348s) [noise=true , address=145.79.227.31:4799 , piece_id=JEBR7PFHJBN4UOTVN5HM3NNZ5WX4IWZNBRUFFJ3QSDSTONAFAR7Q, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.301898620 (595h12m39.172981665s) [noise=true , address=145.79.227.21:4475 , piece_id=WP7NRZXGIFRCI3JVCVCXTVXEFZ7BYZQN5JXLTREWXSKD6ULZYSVQ, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.305326327 (595h12m39.176409362s) [noise=true , address=145.79.227.50:4374 , piece_id=G5Y7BV34YTPBC7QK3A5OVO3P7ILJT5WYOOXZBAA6QJXPXVCK5UXQ, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.305621429 (595h12m39.176704474s) [noise=true , address=145.79.227.45:4229 , piece_id=ZNC76TIUYUXEMZC5FY22U365CICRNFQCSLIXPVZDBF2KNZ2KDI7Q, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.313049473 (595h12m39.184132508s) [noise=true , address=145.79.227.48:4303 , piece_id=CXP7CRYIP4NUZYMTYLRA4N4YLZHSOFOK5B5RF7JXCCAQMTX2GHLA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.313096753 (595h12m39.184179798s) [noise=true , address=145.79.227.25:4619 , piece_id=QPAQFNMX4S7OY5YYUXQQKUG7UYHCXBDSKBL6A7S45OQTY6BYWTNA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.330307055 (595h12m39.20139009s) [noise=true , address=145.79.227.17:4970 , piece_id=VFVP5IJDEG4OUH6H6QHDCKNORBC5US4ZG3273FPBFT7BO3DNYNMA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.331102860 (595h12m39.202185895s) [noise=true , address=145.79.227.11:4730 , piece_id=QMESJWSNK2KAIDZBQWU2OUNMSPEZFWGQXU3QVNFBVEQXYWHPLVXA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.333242113 (595h12m39.204325158s) [noise=true , address=145.79.227.19:4077 , piece_id=4CRE76JFUMHITF7OXCAIC455F7L5SJC5ZCDMDSIJJ6AQ4HSK7AMA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.355266472 (595h12m39.226349507s) [noise=true , address=145.79.227.20:4439 , piece_id=QKDITA4DSTKAIU2S2JXG5H7JM5FH2JE7CDUHAMLMYVPEGF53VOSQ, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.387551277 (595h12m39.258634312s) [noise=true , address=145.79.227.39:4014 , piece_id=Q7CWPW2G3K2O5NZVZQTLZAXJRZG5XYFZZJNP4SHVEIH6RW6CAUOA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.876119214 (595h12m39.747202249s) [noise=true , address=77.90.31.16:4735 , piece_id=KVNNUQGW7FKC5F77RAFEFUFSMXZ62FT7RIVO5F6JFZBCGD7YC4TA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:40.191523548 (595h12m40.062606593s) [noise=true , address=77.90.31.18:4808 , piece_id=6OIBW7HMNQR3UXQYWFDP6J2AZDSL35VO4Z6PJ5P7UCRIJFALL77A, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:40.191549366 (595h12m40.062632401s) [seg_pos={0 0}, upload=521763]: Segment reached optimal threshold of 13 pieces. Dec 30 10:41:40.191708269 (595h12m40.062791304s) [noise=true , address=77.90.31.11:4554 , piece_id=UUJNARWXIOJM775GZKLOXT4ZHBUYDHVTI5STPK2Z7UHYSTGIOV3A, seg_pos={0 0}, upload=521763]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:35868->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:35868->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35868->77.90.31.11:4554: use of closed network connection Dec 30 10:41:40.191714180 (595h12m40.062797225s) [noise=true , address=77.90.31.11:4554 , piece_id=UUJNARWXIOJM775GZKLOXT4ZHBUYDHVTI5STPK2Z7UHYSTGIOV3A, seg_pos={0 0}, upload=521763]: Long tail cancellation due to optimal threshold Dec 30 10:41:40.191795394 (595h12m40.062878459s) [noise=true , address=77.90.31.13:4629 , piece_id=SNJ4Z5D4QNHBSNDLSH7W6NROV3X3YGQYB2MNAIKURRUBSTCNIGOA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 30 10:41:40.191814181 (595h12m40.062897236s) [noise=true , address=77.90.31.13:4629 , piece_id=SNJ4Z5D4QNHBSNDLSH7W6NROV3X3YGQYB2MNAIKURRUBSTCNIGOA, seg_pos={0 0}, upload=521763]: Long tail cancellation due to optimal threshold Dec 30 10:41:40.191917547 (595h12m40.063000583s) [noise=true , address=45.91.116.12:4225 , piece_id=7H55Y5PZTFNLY4SNJDXD2PEARFNP6OXRJJGXRDA2UMPM435B4K3A, seg_pos={0 0}, upload=521763]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 30 10:41:40.191923709 (595h12m40.063006744s) [noise=true , address=45.91.116.12:4225 , piece_id=7H55Y5PZTFNLY4SNJDXD2PEARFNP6OXRJJGXRDA2UMPM435B4K3A, seg_pos={0 0}, upload=521763]: Long tail cancellation due to optimal threshold Dec 30 10:41:40.191939890 (595h12m40.063022935s) [seg_pos={0 0}, upload=521763]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:41:40.192001908 (595h12m40.063084943s) [upload=521763]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:41:40.192006397 (595h12m40.063089442s) [upload=521763]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:41:40.486537746 (595h12m40.357620801s) [upload=521763]: Done uploading segments... Dec 30 10:41:40.486564095 (595h12m40.35764715s) [upload=521763]: Upload finished. err: <nil> Duration: 5.746382616s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [....X...X...X........X.....X.X............X......X.X.XXXX.X.] duration: 4.509307623s trace: https://jaeger.dev.storj.io/trace/4478c63e695ec5b6 FAIL (2025-12-30 10:42:45 UTC) orbiter warning: context deadline exceeded Duration: 4.509339885s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................X....X.........X..................] duration: 38.316666102s trace: https://jaeger.dev.storj.io/trace/1dc8263b2e55d7dc FAIL (2025-12-30 10:42:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311300688 (594h58m28.182383733s) [noise=true , address=145.79.227.43:4179 , piece_id=CQ32O4X3PKQN37OWC3WU5Z5FF7CCAIOAWLUISOXJTTIXTSMKYNQA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311642950 (594h58m28.182725995s) [noise=true , address=145.79.227.30:4783 , piece_id=UIXZYYXWC7KJZUV3BNZAIUSZNDEYA5RFFLKMXWF3XLNKJF5K2A6Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311712022 (594h58m28.182795057s) [noise=true , address=145.79.227.35:4492 , piece_id=GM7TCP65VSFAGWDCRG4VKWAPJHU5HGZFZTVAD4XM6PJFMLCC4NRA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312038483 (594h58m28.183121528s) [noise=true , address=77.90.31.17:4798 , piece_id=HT7E2VVRR2IURW5OIXQ4BLR55QI2BEKJG3VIRG4CABVPCUZRIGWA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312100291 (594h58m28.183183326s) [noise=true , address=77.90.31.19:4859 , piece_id=HEZMENV33Y6A6MMMMDIUECW44662O24WTIG2RWCWZL3IGNKWCH2A, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312494201 (594h58m28.183577246s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312500252 (594h58m28.183583287s) [noise=true , address=145.79.227.49:4359 , piece_id=RGCUJ32JS6NXVVLH7HWUD5IZNV3HZPRRVIHEBJEII2JPEENVIKZA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312899192 (594h58m28.183982227s) [noise=true , address=145.79.227.18:4037 , piece_id=IQZILX3EVK57WA5QGC4VGRYTYWQX37PLIASYIYXIHWBWKGTSEESA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313265008 (594h58m28.184348053s) [noise=true , address=145.79.227.6:4267 , piece_id=ZE5IVJYY2XEW3QOZBBE7LYICAALYG3DZHU6K7UDYUGBHRTWPQYPA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313640203 (594h58m28.184723248s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313904957 (594h58m28.184988012s) [noise=true , address=45.91.116.14:4213 , piece_id=BUTDQ3PI56Q7ZHNYHCEMWPHJUI42LXBR42ZGF4VKMDXJ47BSPU3Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314038000 (594h58m28.185121045s) [noise=true , address=77.90.31.12:4603 , piece_id=QQ5V6RXCHOHVL2QTWLLRFPJBAQQ5KW47T2N44EBMIUW747Q4ZOAA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314410920 (594h58m28.185493965s) [noise=true , address=145.79.227.44:4219 , piece_id=NSHMZDOI264JBC5ML4GRTH66FKT76C5QQ534UA4QDFEIG5P6CHDQ, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314664292 (594h58m28.185747338s) [noise=true , address=145.79.227.40:4062 , piece_id=RTSWJJIEADDCF5Z4JMT4NERSF7EXCWP7QZKDN4KIWEX54UDLNZEA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.315164305 (594h58m28.18624734s) [noise=true , address=77.90.31.10:4540 , piece_id=5LXEPFE5MNCGS6KJK4RP5Q5MV4M3TQMVR5FK4DYTYY4MKTFXTP2Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:29.778852769 (594h58m29.649935804s) [upload=521557]: Next returned nil segment Dec 30 10:27:29.778904818 (594h58m29.649987863s) [upload=521557]: Waiting for error group managing segments... Dec 30 10:27:30.177976438 (594h58m30.049059474s) [noise=true , address=145.79.227.18:4037 , piece_id=IQZILX3EVK57WA5QGC4VGRYTYWQX37PLIASYIYXIHWBWKGTSEESA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.239401413 (594h58m30.110484448s) [noise=true , address=145.79.227.6:4267 , piece_id=ZE5IVJYY2XEW3QOZBBE7LYICAALYG3DZHU6K7UDYUGBHRTWPQYPA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.247231673 (594h58m30.118314708s) [noise=true , address=145.79.227.35:4492 , piece_id=GM7TCP65VSFAGWDCRG4VKWAPJHU5HGZFZTVAD4XM6PJFMLCC4NRA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.364460186 (594h58m30.235543221s) [noise=true , address=77.90.31.17:4798 , piece_id=HT7E2VVRR2IURW5OIXQ4BLR55QI2BEKJG3VIRG4CABVPCUZRIGWA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.415614833 (594h58m30.286697868s) [noise=true , address=77.90.31.12:4603 , piece_id=QQ5V6RXCHOHVL2QTWLLRFPJBAQQ5KW47T2N44EBMIUW747Q4ZOAA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.444886882 (594h58m30.315969917s) [noise=true , address=77.90.31.10:4540 , piece_id=5LXEPFE5MNCGS6KJK4RP5Q5MV4M3TQMVR5FK4DYTYY4MKTFXTP2Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.512047427 (594h58m30.383130462s) [noise=true , address=45.91.116.14:4213 , piece_id=BUTDQ3PI56Q7ZHNYHCEMWPHJUI42LXBR42ZGF4VKMDXJ47BSPU3Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.536847771 (594h58m30.407930806s) [noise=true , address=77.90.31.19:4859 , piece_id=HEZMENV33Y6A6MMMMDIUECW44662O24WTIG2RWCWZL3IGNKWCH2A, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.680580563 (594h58m30.551663608s) [noise=true , address=45.91.116.12:4253 , piece_id=NPLIO7NLEIRTQAWDQ7WZVZE5ZSRLDPDNM34M6SEI53G5XEHB3LJA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.065501378 (594h58m30.936584414s) [noise=true , address=145.79.227.49:4359 , piece_id=RGCUJ32JS6NXVVLH7HWUD5IZNV3HZPRRVIHEBJEII2JPEENVIKZA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.104555514 (594h58m30.975638559s) [noise=true , address=145.79.227.44:4219 , piece_id=NSHMZDOI264JBC5ML4GRTH66FKT76C5QQ534UA4QDFEIG5P6CHDQ, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.115841635 (594h58m30.98692467s) [noise=true , address=145.79.227.30:4783 , piece_id=UIXZYYXWC7KJZUV3BNZAIUSZNDEYA5RFFLKMXWF3XLNKJF5K2A6Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.364792710 (594h58m31.235875755s) [noise=true , address=145.79.227.40:4062 , piece_id=RTSWJJIEADDCF5Z4JMT4NERSF7EXCWP7QZKDN4KIWEX54UDLNZEA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.374552906 (594h58m31.245635941s) [noise=true , address=145.79.227.43:4179 , piece_id=CQ32O4X3PKQN37OWC3WU5Z5FF7CCAIOAWLUISOXJTTIXTSMKYNQA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.374572984 (594h58m31.245656019s) [seg_pos={0 0}, upload=521557]: Segment reached optimal threshold of 14 pieces. Dec 30 10:27:31.374775900 (594h58m31.245858935s) [noise=true , address=45.91.116.10:4115 , piece_id=DRZ2YOFTPSE7SQD6MJ6YQWRQN6MDOLHHP7LWJMKMCPHO37A72C4A, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1arcuTW5rwnvfiqi2H7dL3oL6jKBkDeGq6DmgRGpS9Vs9BCHmf): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.374782553 (594h58m31.245865598s) [noise=true , address=45.91.116.10:4115 , piece_id=DRZ2YOFTPSE7SQD6MJ6YQWRQN6MDOLHHP7LWJMKMCPHO37A72C4A, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.374972405 (594h58m31.24605544s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.374978286 (594h58m31.246061321s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375019294 (594h58m31.246102349s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.375038921 (594h58m31.246121966s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375091411 (594h58m31.246174446s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE): protocol: write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection; write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection Dec 30 10:27:31.375099606 (594h58m31.246182641s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375195900 (594h58m31.246278935s) [seg_pos={0 0}, upload=521557]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:27:31.375261705 (594h58m31.24634474s) [upload=521557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:27:31.375266604 (594h58m31.246349639s) [upload=521557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:27:31.664767884 (594h58m31.53585099s) [upload=521557]: Done uploading segments... Dec 30 10:27:31.664793884 (594h58m31.535876929s) [upload=521557]: Upload finished. err: <nil> Duration: 38.316698273s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX.X.XX.X.X.X.XX.X.....XX..........XX...X.X..XX.XX........X.] duration: 5.819742163s trace: https://jaeger.dev.storj.io/trace/5c4f8766c61bbc8a FAIL (2025-12-30 10:42:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: 6OIBW7HMNQR3UXQYWFDP6J2AZDSL35VO4Z6PJ5P7UCRIJFALL77A) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: KVNNUQGW7FKC5F77RAFEFUFSMXZ62FT7RIVO5F6JFZBCGD7YC4TA) context deadline exceeded ======LOGS====== 0 10:41:37.067033916 (595h12m36.938116961s) [upload=521763]: Starting upload Dec 30 10:41:37.067118447 (595h12m36.938201492s) [upload=521763]: Done starting upload Dec 30 10:41:37.067361941 (595h12m36.938444976s) [upload=521763]: Uploading segments... Dec 30 10:41:37.067504051 (595h12m36.938587096s) [upload=521763]: Got next segment. Inline: false Dec 30 10:41:37.067575377 (595h12m36.938658412s) [upload=521763]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:41:37.067583141 (595h12m36.938666186s) [upload=521763]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:41:37.315237828 (595h12m37.186320863s) [seg_pos={0 0}, upload=521763]: Begin upload segment... Dec 30 10:41:37.315281842 (595h12m37.186364877s) [seg_pos={0 0}, upload=521763]: Done begin upload segment. Dec 30 10:41:37.315306599 (595h12m37.186389644s) [noise=true , address=77.90.31.13:4629 , piece_id=SNJ4Z5D4QNHBSNDLSH7W6NROV3X3YGQYB2MNAIKURRUBSTCNIGOA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.315384688 (595h12m37.186467723s) [noise=true , address=145.79.227.20:4439 , piece_id=QKDITA4DSTKAIU2S2JXG5H7JM5FH2JE7CDUHAMLMYVPEGF53VOSQ, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.315849714 (595h12m37.186932759s) [noise=true , address=145.79.227.25:4619 , piece_id=QPAQFNMX4S7OY5YYUXQQKUG7UYHCXBDSKBL6A7S45OQTY6BYWTNA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.316276256 (595h12m37.187359291s) [noise=true , address=77.90.31.16:4735 , piece_id=KVNNUQGW7FKC5F77RAFEFUFSMXZ62FT7RIVO5F6JFZBCGD7YC4TA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.316667430 (595h12m37.187750476s) [noise=true , address=145.79.227.48:4303 , piece_id=CXP7CRYIP4NUZYMTYLRA4N4YLZHSOFOK5B5RF7JXCCAQMTX2GHLA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.317060378 (595h12m37.188143434s) [noise=true , address=145.79.227.19:4077 , piece_id=4CRE76JFUMHITF7OXCAIC455F7L5SJC5ZCDMDSIJJ6AQ4HSK7AMA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.317419994 (595h12m37.188503029s) [noise=true , address=45.91.116.12:4225 , piece_id=7H55Y5PZTFNLY4SNJDXD2PEARFNP6OXRJJGXRDA2UMPM435B4K3A, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.317948861 (595h12m37.189031896s) [noise=true , address=145.79.227.39:4014 , piece_id=Q7CWPW2G3K2O5NZVZQTLZAXJRZG5XYFZZJNP4SHVEIH6RW6CAUOA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318277436 (595h12m37.189360481s) [noise=true , address=145.79.227.17:4970 , piece_id=VFVP5IJDEG4OUH6H6QHDCKNORBC5US4ZG3273FPBFT7BO3DNYNMA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318313034 (595h12m37.189396069s) [noise=true , address=145.79.227.45:4229 , piece_id=ZNC76TIUYUXEMZC5FY22U365CICRNFQCSLIXPVZDBF2KNZ2KDI7Q, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318343201 (595h12m37.189426236s) [noise=true , address=145.79.227.21:4475 , piece_id=WP7NRZXGIFRCI3JVCVCXTVXEFZ7BYZQN5JXLTREWXSKD6ULZYSVQ, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318616161 (595h12m37.189699196s) [noise=true , address=145.79.227.11:4730 , piece_id=QMESJWSNK2KAIDZBQWU2OUNMSPEZFWGQXU3QVNFBVEQXYWHPLVXA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318826802 (595h12m37.189909837s) [noise=true , address=145.79.227.50:4374 , piece_id=G5Y7BV34YTPBC7QK3A5OVO3P7ILJT5WYOOXZBAA6QJXPXVCK5UXQ, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318975465 (595h12m37.19005851s) [noise=true , address=145.79.227.31:4799 , piece_id=JEBR7PFHJBN4UOTVN5HM3NNZ5WX4IWZNBRUFFJ3QSDSTONAFAR7Q, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.319245860 (595h12m37.190328895s) [noise=true , address=77.90.31.11:4554 , piece_id=UUJNARWXIOJM775GZKLOXT4ZHBUYDHVTI5STPK2Z7UHYSTGIOV3A, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.321923038 (595h12m37.193006083s) [noise=true , address=77.90.31.18:4808 , piece_id=6OIBW7HMNQR3UXQYWFDP6J2AZDSL35VO4Z6PJ5P7UCRIJFALL77A, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:38.919914306 (595h12m38.790997351s) [upload=521763]: Next returned nil segment Dec 30 10:41:38.919944133 (595h12m38.791027178s) [upload=521763]: Waiting for error group managing segments... Dec 30 10:41:39.299400313 (595h12m39.170483348s) [noise=true , address=145.79.227.31:4799 , piece_id=JEBR7PFHJBN4UOTVN5HM3NNZ5WX4IWZNBRUFFJ3QSDSTONAFAR7Q, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.301898620 (595h12m39.172981665s) [noise=true , address=145.79.227.21:4475 , piece_id=WP7NRZXGIFRCI3JVCVCXTVXEFZ7BYZQN5JXLTREWXSKD6ULZYSVQ, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.305326327 (595h12m39.176409362s) [noise=true , address=145.79.227.50:4374 , piece_id=G5Y7BV34YTPBC7QK3A5OVO3P7ILJT5WYOOXZBAA6QJXPXVCK5UXQ, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.305621429 (595h12m39.176704474s) [noise=true , address=145.79.227.45:4229 , piece_id=ZNC76TIUYUXEMZC5FY22U365CICRNFQCSLIXPVZDBF2KNZ2KDI7Q, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.313049473 (595h12m39.184132508s) [noise=true , address=145.79.227.48:4303 , piece_id=CXP7CRYIP4NUZYMTYLRA4N4YLZHSOFOK5B5RF7JXCCAQMTX2GHLA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.313096753 (595h12m39.184179798s) [noise=true , address=145.79.227.25:4619 , piece_id=QPAQFNMX4S7OY5YYUXQQKUG7UYHCXBDSKBL6A7S45OQTY6BYWTNA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.330307055 (595h12m39.20139009s) [noise=true , address=145.79.227.17:4970 , piece_id=VFVP5IJDEG4OUH6H6QHDCKNORBC5US4ZG3273FPBFT7BO3DNYNMA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.331102860 (595h12m39.202185895s) [noise=true , address=145.79.227.11:4730 , piece_id=QMESJWSNK2KAIDZBQWU2OUNMSPEZFWGQXU3QVNFBVEQXYWHPLVXA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.333242113 (595h12m39.204325158s) [noise=true , address=145.79.227.19:4077 , piece_id=4CRE76JFUMHITF7OXCAIC455F7L5SJC5ZCDMDSIJJ6AQ4HSK7AMA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.355266472 (595h12m39.226349507s) [noise=true , address=145.79.227.20:4439 , piece_id=QKDITA4DSTKAIU2S2JXG5H7JM5FH2JE7CDUHAMLMYVPEGF53VOSQ, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.387551277 (595h12m39.258634312s) [noise=true , address=145.79.227.39:4014 , piece_id=Q7CWPW2G3K2O5NZVZQTLZAXJRZG5XYFZZJNP4SHVEIH6RW6CAUOA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.876119214 (595h12m39.747202249s) [noise=true , address=77.90.31.16:4735 , piece_id=KVNNUQGW7FKC5F77RAFEFUFSMXZ62FT7RIVO5F6JFZBCGD7YC4TA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:40.191523548 (595h12m40.062606593s) [noise=true , address=77.90.31.18:4808 , piece_id=6OIBW7HMNQR3UXQYWFDP6J2AZDSL35VO4Z6PJ5P7UCRIJFALL77A, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:40.191549366 (595h12m40.062632401s) [seg_pos={0 0}, upload=521763]: Segment reached optimal threshold of 13 pieces. Dec 30 10:41:40.191708269 (595h12m40.062791304s) [noise=true , address=77.90.31.11:4554 , piece_id=UUJNARWXIOJM775GZKLOXT4ZHBUYDHVTI5STPK2Z7UHYSTGIOV3A, seg_pos={0 0}, upload=521763]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:35868->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:35868->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35868->77.90.31.11:4554: use of closed network connection Dec 30 10:41:40.191714180 (595h12m40.062797225s) [noise=true , address=77.90.31.11:4554 , piece_id=UUJNARWXIOJM775GZKLOXT4ZHBUYDHVTI5STPK2Z7UHYSTGIOV3A, seg_pos={0 0}, upload=521763]: Long tail cancellation due to optimal threshold Dec 30 10:41:40.191795394 (595h12m40.062878459s) [noise=true , address=77.90.31.13:4629 , piece_id=SNJ4Z5D4QNHBSNDLSH7W6NROV3X3YGQYB2MNAIKURRUBSTCNIGOA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 30 10:41:40.191814181 (595h12m40.062897236s) [noise=true , address=77.90.31.13:4629 , piece_id=SNJ4Z5D4QNHBSNDLSH7W6NROV3X3YGQYB2MNAIKURRUBSTCNIGOA, seg_pos={0 0}, upload=521763]: Long tail cancellation due to optimal threshold Dec 30 10:41:40.191917547 (595h12m40.063000583s) [noise=true , address=45.91.116.12:4225 , piece_id=7H55Y5PZTFNLY4SNJDXD2PEARFNP6OXRJJGXRDA2UMPM435B4K3A, seg_pos={0 0}, upload=521763]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 30 10:41:40.191923709 (595h12m40.063006744s) [noise=true , address=45.91.116.12:4225 , piece_id=7H55Y5PZTFNLY4SNJDXD2PEARFNP6OXRJJGXRDA2UMPM435B4K3A, seg_pos={0 0}, upload=521763]: Long tail cancellation due to optimal threshold Dec 30 10:41:40.191939890 (595h12m40.063022935s) [seg_pos={0 0}, upload=521763]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:41:40.192001908 (595h12m40.063084943s) [upload=521763]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:41:40.192006397 (595h12m40.063089442s) [upload=521763]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:41:40.486537746 (595h12m40.357620801s) [upload=521763]: Done uploading segments... Dec 30 10:41:40.486564095 (595h12m40.35764715s) [upload=521763]: Upload finished. err: <nil> Duration: 5.819766761s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XX.X.XX.X.X.X.XX.X.....XX..........XX...X.X..XX.XX........X] duration: 30.002934396s trace: https://jaeger.dev.storj.io/trace/5c4f8766c61bbc8a FAIL (2025-12-30 10:42:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: 6OIBW7HMNQR3UXQYWFDP6J2AZDSL35VO4Z6PJ5P7UCRIJFALL77A) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: KVNNUQGW7FKC5F77RAFEFUFSMXZ62FT7RIVO5F6JFZBCGD7YC4TA) context deadline exceeded ======LOGS====== 0 10:41:37.067033916 (595h12m36.938116961s) [upload=521763]: Starting upload Dec 30 10:41:37.067118447 (595h12m36.938201492s) [upload=521763]: Done starting upload Dec 30 10:41:37.067361941 (595h12m36.938444976s) [upload=521763]: Uploading segments... Dec 30 10:41:37.067504051 (595h12m36.938587096s) [upload=521763]: Got next segment. Inline: false Dec 30 10:41:37.067575377 (595h12m36.938658412s) [upload=521763]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:41:37.067583141 (595h12m36.938666186s) [upload=521763]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:41:37.315237828 (595h12m37.186320863s) [seg_pos={0 0}, upload=521763]: Begin upload segment... Dec 30 10:41:37.315281842 (595h12m37.186364877s) [seg_pos={0 0}, upload=521763]: Done begin upload segment. Dec 30 10:41:37.315306599 (595h12m37.186389644s) [noise=true , address=77.90.31.13:4629 , piece_id=SNJ4Z5D4QNHBSNDLSH7W6NROV3X3YGQYB2MNAIKURRUBSTCNIGOA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.315384688 (595h12m37.186467723s) [noise=true , address=145.79.227.20:4439 , piece_id=QKDITA4DSTKAIU2S2JXG5H7JM5FH2JE7CDUHAMLMYVPEGF53VOSQ, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.315849714 (595h12m37.186932759s) [noise=true , address=145.79.227.25:4619 , piece_id=QPAQFNMX4S7OY5YYUXQQKUG7UYHCXBDSKBL6A7S45OQTY6BYWTNA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.316276256 (595h12m37.187359291s) [noise=true , address=77.90.31.16:4735 , piece_id=KVNNUQGW7FKC5F77RAFEFUFSMXZ62FT7RIVO5F6JFZBCGD7YC4TA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.316667430 (595h12m37.187750476s) [noise=true , address=145.79.227.48:4303 , piece_id=CXP7CRYIP4NUZYMTYLRA4N4YLZHSOFOK5B5RF7JXCCAQMTX2GHLA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.317060378 (595h12m37.188143434s) [noise=true , address=145.79.227.19:4077 , piece_id=4CRE76JFUMHITF7OXCAIC455F7L5SJC5ZCDMDSIJJ6AQ4HSK7AMA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.317419994 (595h12m37.188503029s) [noise=true , address=45.91.116.12:4225 , piece_id=7H55Y5PZTFNLY4SNJDXD2PEARFNP6OXRJJGXRDA2UMPM435B4K3A, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.317948861 (595h12m37.189031896s) [noise=true , address=145.79.227.39:4014 , piece_id=Q7CWPW2G3K2O5NZVZQTLZAXJRZG5XYFZZJNP4SHVEIH6RW6CAUOA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318277436 (595h12m37.189360481s) [noise=true , address=145.79.227.17:4970 , piece_id=VFVP5IJDEG4OUH6H6QHDCKNORBC5US4ZG3273FPBFT7BO3DNYNMA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318313034 (595h12m37.189396069s) [noise=true , address=145.79.227.45:4229 , piece_id=ZNC76TIUYUXEMZC5FY22U365CICRNFQCSLIXPVZDBF2KNZ2KDI7Q, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318343201 (595h12m37.189426236s) [noise=true , address=145.79.227.21:4475 , piece_id=WP7NRZXGIFRCI3JVCVCXTVXEFZ7BYZQN5JXLTREWXSKD6ULZYSVQ, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318616161 (595h12m37.189699196s) [noise=true , address=145.79.227.11:4730 , piece_id=QMESJWSNK2KAIDZBQWU2OUNMSPEZFWGQXU3QVNFBVEQXYWHPLVXA, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318826802 (595h12m37.189909837s) [noise=true , address=145.79.227.50:4374 , piece_id=G5Y7BV34YTPBC7QK3A5OVO3P7ILJT5WYOOXZBAA6QJXPXVCK5UXQ, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.318975465 (595h12m37.19005851s) [noise=true , address=145.79.227.31:4799 , piece_id=JEBR7PFHJBN4UOTVN5HM3NNZ5WX4IWZNBRUFFJ3QSDSTONAFAR7Q, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.319245860 (595h12m37.190328895s) [noise=true , address=77.90.31.11:4554 , piece_id=UUJNARWXIOJM775GZKLOXT4ZHBUYDHVTI5STPK2Z7UHYSTGIOV3A, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:37.321923038 (595h12m37.193006083s) [noise=true , address=77.90.31.18:4808 , piece_id=6OIBW7HMNQR3UXQYWFDP6J2AZDSL35VO4Z6PJ5P7UCRIJFALL77A, seg_pos={0 0}, upload=521763]: Uploading piece... Dec 30 10:41:38.919914306 (595h12m38.790997351s) [upload=521763]: Next returned nil segment Dec 30 10:41:38.919944133 (595h12m38.791027178s) [upload=521763]: Waiting for error group managing segments... Dec 30 10:41:39.299400313 (595h12m39.170483348s) [noise=true , address=145.79.227.31:4799 , piece_id=JEBR7PFHJBN4UOTVN5HM3NNZ5WX4IWZNBRUFFJ3QSDSTONAFAR7Q, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.301898620 (595h12m39.172981665s) [noise=true , address=145.79.227.21:4475 , piece_id=WP7NRZXGIFRCI3JVCVCXTVXEFZ7BYZQN5JXLTREWXSKD6ULZYSVQ, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.305326327 (595h12m39.176409362s) [noise=true , address=145.79.227.50:4374 , piece_id=G5Y7BV34YTPBC7QK3A5OVO3P7ILJT5WYOOXZBAA6QJXPXVCK5UXQ, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.305621429 (595h12m39.176704474s) [noise=true , address=145.79.227.45:4229 , piece_id=ZNC76TIUYUXEMZC5FY22U365CICRNFQCSLIXPVZDBF2KNZ2KDI7Q, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.313049473 (595h12m39.184132508s) [noise=true , address=145.79.227.48:4303 , piece_id=CXP7CRYIP4NUZYMTYLRA4N4YLZHSOFOK5B5RF7JXCCAQMTX2GHLA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.313096753 (595h12m39.184179798s) [noise=true , address=145.79.227.25:4619 , piece_id=QPAQFNMX4S7OY5YYUXQQKUG7UYHCXBDSKBL6A7S45OQTY6BYWTNA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.330307055 (595h12m39.20139009s) [noise=true , address=145.79.227.17:4970 , piece_id=VFVP5IJDEG4OUH6H6QHDCKNORBC5US4ZG3273FPBFT7BO3DNYNMA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.331102860 (595h12m39.202185895s) [noise=true , address=145.79.227.11:4730 , piece_id=QMESJWSNK2KAIDZBQWU2OUNMSPEZFWGQXU3QVNFBVEQXYWHPLVXA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.333242113 (595h12m39.204325158s) [noise=true , address=145.79.227.19:4077 , piece_id=4CRE76JFUMHITF7OXCAIC455F7L5SJC5ZCDMDSIJJ6AQ4HSK7AMA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.355266472 (595h12m39.226349507s) [noise=true , address=145.79.227.20:4439 , piece_id=QKDITA4DSTKAIU2S2JXG5H7JM5FH2JE7CDUHAMLMYVPEGF53VOSQ, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.387551277 (595h12m39.258634312s) [noise=true , address=145.79.227.39:4014 , piece_id=Q7CWPW2G3K2O5NZVZQTLZAXJRZG5XYFZZJNP4SHVEIH6RW6CAUOA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:39.876119214 (595h12m39.747202249s) [noise=true , address=77.90.31.16:4735 , piece_id=KVNNUQGW7FKC5F77RAFEFUFSMXZ62FT7RIVO5F6JFZBCGD7YC4TA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:40.191523548 (595h12m40.062606593s) [noise=true , address=77.90.31.18:4808 , piece_id=6OIBW7HMNQR3UXQYWFDP6J2AZDSL35VO4Z6PJ5P7UCRIJFALL77A, seg_pos={0 0}, upload=521763]: Done uploading piece. err: <nil> Dec 30 10:41:40.191549366 (595h12m40.062632401s) [seg_pos={0 0}, upload=521763]: Segment reached optimal threshold of 13 pieces. Dec 30 10:41:40.191708269 (595h12m40.062791304s) [noise=true , address=77.90.31.11:4554 , piece_id=UUJNARWXIOJM775GZKLOXT4ZHBUYDHVTI5STPK2Z7UHYSTGIOV3A, seg_pos={0 0}, upload=521763]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:35868->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:35868->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35868->77.90.31.11:4554: use of closed network connection Dec 30 10:41:40.191714180 (595h12m40.062797225s) [noise=true , address=77.90.31.11:4554 , piece_id=UUJNARWXIOJM775GZKLOXT4ZHBUYDHVTI5STPK2Z7UHYSTGIOV3A, seg_pos={0 0}, upload=521763]: Long tail cancellation due to optimal threshold Dec 30 10:41:40.191795394 (595h12m40.062878459s) [noise=true , address=77.90.31.13:4629 , piece_id=SNJ4Z5D4QNHBSNDLSH7W6NROV3X3YGQYB2MNAIKURRUBSTCNIGOA, seg_pos={0 0}, upload=521763]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 30 10:41:40.191814181 (595h12m40.062897236s) [noise=true , address=77.90.31.13:4629 , piece_id=SNJ4Z5D4QNHBSNDLSH7W6NROV3X3YGQYB2MNAIKURRUBSTCNIGOA, seg_pos={0 0}, upload=521763]: Long tail cancellation due to optimal threshold Dec 30 10:41:40.191917547 (595h12m40.063000583s) [noise=true , address=45.91.116.12:4225 , piece_id=7H55Y5PZTFNLY4SNJDXD2PEARFNP6OXRJJGXRDA2UMPM435B4K3A, seg_pos={0 0}, upload=521763]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 30 10:41:40.191923709 (595h12m40.063006744s) [noise=true , address=45.91.116.12:4225 , piece_id=7H55Y5PZTFNLY4SNJDXD2PEARFNP6OXRJJGXRDA2UMPM435B4K3A, seg_pos={0 0}, upload=521763]: Long tail cancellation due to optimal threshold Dec 30 10:41:40.191939890 (595h12m40.063022935s) [seg_pos={0 0}, upload=521763]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:41:40.192001908 (595h12m40.063084943s) [upload=521763]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:41:40.192006397 (595h12m40.063089442s) [upload=521763]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:41:40.486537746 (595h12m40.357620801s) [upload=521763]: Done uploading segments... Dec 30 10:41:40.486564095 (595h12m40.35764715s) [upload=521763]: Upload finished. err: <nil> Duration: 30.002961978s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.....X...X...X........X.....X.X............X......X.X.XXXX.X] duration: 5.000862456s trace: https://jaeger.dev.storj.io/trace/4478c63e695ec5b6 FAIL (2025-12-30 10:42:05 UTC) orbiter warning: context deadline exceeded Duration: 5.000883997s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [......X...X...X........X.....X.X............X......X.X.XXXX.] duration: 3.675220067s trace: https://jaeger.dev.storj.io/trace/29e0480a91fb5227 FAIL (2025-12-30 10:41:38 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.675236939s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................X....X.........X.................] duration: 15.463077351s trace: https://jaeger.dev.storj.io/trace/1dc8263b2e55d7dc FAIL (2025-12-30 10:41:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311300688 (594h58m28.182383733s) [noise=true , address=145.79.227.43:4179 , piece_id=CQ32O4X3PKQN37OWC3WU5Z5FF7CCAIOAWLUISOXJTTIXTSMKYNQA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311642950 (594h58m28.182725995s) [noise=true , address=145.79.227.30:4783 , piece_id=UIXZYYXWC7KJZUV3BNZAIUSZNDEYA5RFFLKMXWF3XLNKJF5K2A6Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311712022 (594h58m28.182795057s) [noise=true , address=145.79.227.35:4492 , piece_id=GM7TCP65VSFAGWDCRG4VKWAPJHU5HGZFZTVAD4XM6PJFMLCC4NRA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312038483 (594h58m28.183121528s) [noise=true , address=77.90.31.17:4798 , piece_id=HT7E2VVRR2IURW5OIXQ4BLR55QI2BEKJG3VIRG4CABVPCUZRIGWA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312100291 (594h58m28.183183326s) [noise=true , address=77.90.31.19:4859 , piece_id=HEZMENV33Y6A6MMMMDIUECW44662O24WTIG2RWCWZL3IGNKWCH2A, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312494201 (594h58m28.183577246s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312500252 (594h58m28.183583287s) [noise=true , address=145.79.227.49:4359 , piece_id=RGCUJ32JS6NXVVLH7HWUD5IZNV3HZPRRVIHEBJEII2JPEENVIKZA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312899192 (594h58m28.183982227s) [noise=true , address=145.79.227.18:4037 , piece_id=IQZILX3EVK57WA5QGC4VGRYTYWQX37PLIASYIYXIHWBWKGTSEESA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313265008 (594h58m28.184348053s) [noise=true , address=145.79.227.6:4267 , piece_id=ZE5IVJYY2XEW3QOZBBE7LYICAALYG3DZHU6K7UDYUGBHRTWPQYPA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313640203 (594h58m28.184723248s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313904957 (594h58m28.184988012s) [noise=true , address=45.91.116.14:4213 , piece_id=BUTDQ3PI56Q7ZHNYHCEMWPHJUI42LXBR42ZGF4VKMDXJ47BSPU3Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314038000 (594h58m28.185121045s) [noise=true , address=77.90.31.12:4603 , piece_id=QQ5V6RXCHOHVL2QTWLLRFPJBAQQ5KW47T2N44EBMIUW747Q4ZOAA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314410920 (594h58m28.185493965s) [noise=true , address=145.79.227.44:4219 , piece_id=NSHMZDOI264JBC5ML4GRTH66FKT76C5QQ534UA4QDFEIG5P6CHDQ, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314664292 (594h58m28.185747338s) [noise=true , address=145.79.227.40:4062 , piece_id=RTSWJJIEADDCF5Z4JMT4NERSF7EXCWP7QZKDN4KIWEX54UDLNZEA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.315164305 (594h58m28.18624734s) [noise=true , address=77.90.31.10:4540 , piece_id=5LXEPFE5MNCGS6KJK4RP5Q5MV4M3TQMVR5FK4DYTYY4MKTFXTP2Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:29.778852769 (594h58m29.649935804s) [upload=521557]: Next returned nil segment Dec 30 10:27:29.778904818 (594h58m29.649987863s) [upload=521557]: Waiting for error group managing segments... Dec 30 10:27:30.177976438 (594h58m30.049059474s) [noise=true , address=145.79.227.18:4037 , piece_id=IQZILX3EVK57WA5QGC4VGRYTYWQX37PLIASYIYXIHWBWKGTSEESA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.239401413 (594h58m30.110484448s) [noise=true , address=145.79.227.6:4267 , piece_id=ZE5IVJYY2XEW3QOZBBE7LYICAALYG3DZHU6K7UDYUGBHRTWPQYPA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.247231673 (594h58m30.118314708s) [noise=true , address=145.79.227.35:4492 , piece_id=GM7TCP65VSFAGWDCRG4VKWAPJHU5HGZFZTVAD4XM6PJFMLCC4NRA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.364460186 (594h58m30.235543221s) [noise=true , address=77.90.31.17:4798 , piece_id=HT7E2VVRR2IURW5OIXQ4BLR55QI2BEKJG3VIRG4CABVPCUZRIGWA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.415614833 (594h58m30.286697868s) [noise=true , address=77.90.31.12:4603 , piece_id=QQ5V6RXCHOHVL2QTWLLRFPJBAQQ5KW47T2N44EBMIUW747Q4ZOAA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.444886882 (594h58m30.315969917s) [noise=true , address=77.90.31.10:4540 , piece_id=5LXEPFE5MNCGS6KJK4RP5Q5MV4M3TQMVR5FK4DYTYY4MKTFXTP2Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.512047427 (594h58m30.383130462s) [noise=true , address=45.91.116.14:4213 , piece_id=BUTDQ3PI56Q7ZHNYHCEMWPHJUI42LXBR42ZGF4VKMDXJ47BSPU3Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.536847771 (594h58m30.407930806s) [noise=true , address=77.90.31.19:4859 , piece_id=HEZMENV33Y6A6MMMMDIUECW44662O24WTIG2RWCWZL3IGNKWCH2A, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.680580563 (594h58m30.551663608s) [noise=true , address=45.91.116.12:4253 , piece_id=NPLIO7NLEIRTQAWDQ7WZVZE5ZSRLDPDNM34M6SEI53G5XEHB3LJA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.065501378 (594h58m30.936584414s) [noise=true , address=145.79.227.49:4359 , piece_id=RGCUJ32JS6NXVVLH7HWUD5IZNV3HZPRRVIHEBJEII2JPEENVIKZA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.104555514 (594h58m30.975638559s) [noise=true , address=145.79.227.44:4219 , piece_id=NSHMZDOI264JBC5ML4GRTH66FKT76C5QQ534UA4QDFEIG5P6CHDQ, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.115841635 (594h58m30.98692467s) [noise=true , address=145.79.227.30:4783 , piece_id=UIXZYYXWC7KJZUV3BNZAIUSZNDEYA5RFFLKMXWF3XLNKJF5K2A6Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.364792710 (594h58m31.235875755s) [noise=true , address=145.79.227.40:4062 , piece_id=RTSWJJIEADDCF5Z4JMT4NERSF7EXCWP7QZKDN4KIWEX54UDLNZEA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.374552906 (594h58m31.245635941s) [noise=true , address=145.79.227.43:4179 , piece_id=CQ32O4X3PKQN37OWC3WU5Z5FF7CCAIOAWLUISOXJTTIXTSMKYNQA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.374572984 (594h58m31.245656019s) [seg_pos={0 0}, upload=521557]: Segment reached optimal threshold of 14 pieces. Dec 30 10:27:31.374775900 (594h58m31.245858935s) [noise=true , address=45.91.116.10:4115 , piece_id=DRZ2YOFTPSE7SQD6MJ6YQWRQN6MDOLHHP7LWJMKMCPHO37A72C4A, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1arcuTW5rwnvfiqi2H7dL3oL6jKBkDeGq6DmgRGpS9Vs9BCHmf): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.374782553 (594h58m31.245865598s) [noise=true , address=45.91.116.10:4115 , piece_id=DRZ2YOFTPSE7SQD6MJ6YQWRQN6MDOLHHP7LWJMKMCPHO37A72C4A, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.374972405 (594h58m31.24605544s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.374978286 (594h58m31.246061321s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375019294 (594h58m31.246102349s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.375038921 (594h58m31.246121966s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375091411 (594h58m31.246174446s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE): protocol: write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection; write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection Dec 30 10:27:31.375099606 (594h58m31.246182641s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375195900 (594h58m31.246278935s) [seg_pos={0 0}, upload=521557]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:27:31.375261705 (594h58m31.24634474s) [upload=521557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:27:31.375266604 (594h58m31.246349639s) [upload=521557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:27:31.664767884 (594h58m31.53585099s) [upload=521557]: Done uploading segments... Dec 30 10:27:31.664793884 (594h58m31.535876929s) [upload=521557]: Upload finished. err: <nil> Duration: 15.463107999s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................X....X.........X................] duration: 15.216358525s trace: https://jaeger.dev.storj.io/trace/1dc8263b2e55d7dc FAIL (2025-12-30 10:40:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311300688 (594h58m28.182383733s) [noise=true , address=145.79.227.43:4179 , piece_id=CQ32O4X3PKQN37OWC3WU5Z5FF7CCAIOAWLUISOXJTTIXTSMKYNQA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311642950 (594h58m28.182725995s) [noise=true , address=145.79.227.30:4783 , piece_id=UIXZYYXWC7KJZUV3BNZAIUSZNDEYA5RFFLKMXWF3XLNKJF5K2A6Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311712022 (594h58m28.182795057s) [noise=true , address=145.79.227.35:4492 , piece_id=GM7TCP65VSFAGWDCRG4VKWAPJHU5HGZFZTVAD4XM6PJFMLCC4NRA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312038483 (594h58m28.183121528s) [noise=true , address=77.90.31.17:4798 , piece_id=HT7E2VVRR2IURW5OIXQ4BLR55QI2BEKJG3VIRG4CABVPCUZRIGWA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312100291 (594h58m28.183183326s) [noise=true , address=77.90.31.19:4859 , piece_id=HEZMENV33Y6A6MMMMDIUECW44662O24WTIG2RWCWZL3IGNKWCH2A, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312494201 (594h58m28.183577246s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312500252 (594h58m28.183583287s) [noise=true , address=145.79.227.49:4359 , piece_id=RGCUJ32JS6NXVVLH7HWUD5IZNV3HZPRRVIHEBJEII2JPEENVIKZA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312899192 (594h58m28.183982227s) [noise=true , address=145.79.227.18:4037 , piece_id=IQZILX3EVK57WA5QGC4VGRYTYWQX37PLIASYIYXIHWBWKGTSEESA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313265008 (594h58m28.184348053s) [noise=true , address=145.79.227.6:4267 , piece_id=ZE5IVJYY2XEW3QOZBBE7LYICAALYG3DZHU6K7UDYUGBHRTWPQYPA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313640203 (594h58m28.184723248s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313904957 (594h58m28.184988012s) [noise=true , address=45.91.116.14:4213 , piece_id=BUTDQ3PI56Q7ZHNYHCEMWPHJUI42LXBR42ZGF4VKMDXJ47BSPU3Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314038000 (594h58m28.185121045s) [noise=true , address=77.90.31.12:4603 , piece_id=QQ5V6RXCHOHVL2QTWLLRFPJBAQQ5KW47T2N44EBMIUW747Q4ZOAA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314410920 (594h58m28.185493965s) [noise=true , address=145.79.227.44:4219 , piece_id=NSHMZDOI264JBC5ML4GRTH66FKT76C5QQ534UA4QDFEIG5P6CHDQ, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314664292 (594h58m28.185747338s) [noise=true , address=145.79.227.40:4062 , piece_id=RTSWJJIEADDCF5Z4JMT4NERSF7EXCWP7QZKDN4KIWEX54UDLNZEA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.315164305 (594h58m28.18624734s) [noise=true , address=77.90.31.10:4540 , piece_id=5LXEPFE5MNCGS6KJK4RP5Q5MV4M3TQMVR5FK4DYTYY4MKTFXTP2Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:29.778852769 (594h58m29.649935804s) [upload=521557]: Next returned nil segment Dec 30 10:27:29.778904818 (594h58m29.649987863s) [upload=521557]: Waiting for error group managing segments... Dec 30 10:27:30.177976438 (594h58m30.049059474s) [noise=true , address=145.79.227.18:4037 , piece_id=IQZILX3EVK57WA5QGC4VGRYTYWQX37PLIASYIYXIHWBWKGTSEESA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.239401413 (594h58m30.110484448s) [noise=true , address=145.79.227.6:4267 , piece_id=ZE5IVJYY2XEW3QOZBBE7LYICAALYG3DZHU6K7UDYUGBHRTWPQYPA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.247231673 (594h58m30.118314708s) [noise=true , address=145.79.227.35:4492 , piece_id=GM7TCP65VSFAGWDCRG4VKWAPJHU5HGZFZTVAD4XM6PJFMLCC4NRA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.364460186 (594h58m30.235543221s) [noise=true , address=77.90.31.17:4798 , piece_id=HT7E2VVRR2IURW5OIXQ4BLR55QI2BEKJG3VIRG4CABVPCUZRIGWA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.415614833 (594h58m30.286697868s) [noise=true , address=77.90.31.12:4603 , piece_id=QQ5V6RXCHOHVL2QTWLLRFPJBAQQ5KW47T2N44EBMIUW747Q4ZOAA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.444886882 (594h58m30.315969917s) [noise=true , address=77.90.31.10:4540 , piece_id=5LXEPFE5MNCGS6KJK4RP5Q5MV4M3TQMVR5FK4DYTYY4MKTFXTP2Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.512047427 (594h58m30.383130462s) [noise=true , address=45.91.116.14:4213 , piece_id=BUTDQ3PI56Q7ZHNYHCEMWPHJUI42LXBR42ZGF4VKMDXJ47BSPU3Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.536847771 (594h58m30.407930806s) [noise=true , address=77.90.31.19:4859 , piece_id=HEZMENV33Y6A6MMMMDIUECW44662O24WTIG2RWCWZL3IGNKWCH2A, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.680580563 (594h58m30.551663608s) [noise=true , address=45.91.116.12:4253 , piece_id=NPLIO7NLEIRTQAWDQ7WZVZE5ZSRLDPDNM34M6SEI53G5XEHB3LJA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.065501378 (594h58m30.936584414s) [noise=true , address=145.79.227.49:4359 , piece_id=RGCUJ32JS6NXVVLH7HWUD5IZNV3HZPRRVIHEBJEII2JPEENVIKZA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.104555514 (594h58m30.975638559s) [noise=true , address=145.79.227.44:4219 , piece_id=NSHMZDOI264JBC5ML4GRTH66FKT76C5QQ534UA4QDFEIG5P6CHDQ, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.115841635 (594h58m30.98692467s) [noise=true , address=145.79.227.30:4783 , piece_id=UIXZYYXWC7KJZUV3BNZAIUSZNDEYA5RFFLKMXWF3XLNKJF5K2A6Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.364792710 (594h58m31.235875755s) [noise=true , address=145.79.227.40:4062 , piece_id=RTSWJJIEADDCF5Z4JMT4NERSF7EXCWP7QZKDN4KIWEX54UDLNZEA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.374552906 (594h58m31.245635941s) [noise=true , address=145.79.227.43:4179 , piece_id=CQ32O4X3PKQN37OWC3WU5Z5FF7CCAIOAWLUISOXJTTIXTSMKYNQA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.374572984 (594h58m31.245656019s) [seg_pos={0 0}, upload=521557]: Segment reached optimal threshold of 14 pieces. Dec 30 10:27:31.374775900 (594h58m31.245858935s) [noise=true , address=45.91.116.10:4115 , piece_id=DRZ2YOFTPSE7SQD6MJ6YQWRQN6MDOLHHP7LWJMKMCPHO37A72C4A, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1arcuTW5rwnvfiqi2H7dL3oL6jKBkDeGq6DmgRGpS9Vs9BCHmf): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.374782553 (594h58m31.245865598s) [noise=true , address=45.91.116.10:4115 , piece_id=DRZ2YOFTPSE7SQD6MJ6YQWRQN6MDOLHHP7LWJMKMCPHO37A72C4A, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.374972405 (594h58m31.24605544s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.374978286 (594h58m31.246061321s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375019294 (594h58m31.246102349s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.375038921 (594h58m31.246121966s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375091411 (594h58m31.246174446s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE): protocol: write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection; write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection Dec 30 10:27:31.375099606 (594h58m31.246182641s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375195900 (594h58m31.246278935s) [seg_pos={0 0}, upload=521557]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:27:31.375261705 (594h58m31.24634474s) [upload=521557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:27:31.375266604 (594h58m31.246349639s) [upload=521557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:27:31.664767884 (594h58m31.53585099s) [upload=521557]: Done uploading segments... Dec 30 10:27:31.664793884 (594h58m31.535876929s) [upload=521557]: Upload finished. err: <nil> Duration: 15.216388974s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [X......X...X...X........X.....X.X............X......X.X.XXXX] duration: 5.002127035s trace: https://jaeger.dev.storj.io/trace/29e0480a91fb5227 FAIL (2025-12-30 10:40:48 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002148706s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..XX.X.XX.X.X.X.XX.X.....XX..........XX...X.X..XX.XX........] duration: 6.067817561s trace: https://jaeger.dev.storj.io/trace/495530043a5329b3 FAIL (2025-12-30 10:40:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: M7X6D4CJSGVAKXBETA325RFNF7766XPWVVIBUWDTCKPXUW2KU6DQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: 3PAEOVMQUSNSDMHYIHZEIHKRUGCQF3XI3CXYWNYH4MQJX7W3UG3A) context deadline exceeded ======LOGS====== Dec 30 10:35:37.649698347 (595h6m37.520781382s) [upload=521677]: Starting upload Dec 30 10:35:37.649784342 (595h6m37.520867397s) [upload=521677]: Done starting upload Dec 30 10:35:37.649861779 (595h6m37.520944814s) [upload=521677]: Uploading segments... Dec 30 10:35:37.650077299 (595h6m37.521160334s) [upload=521677]: Got next segment. Inline: false Dec 30 10:35:37.650127594 (595h6m37.521210639s) [upload=521677]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:35:37.650133907 (595h6m37.521216952s) [upload=521677]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:35:37.898044742 (595h6m37.769127777s) [seg_pos={0 0}, upload=521677]: Begin upload segment... Dec 30 10:35:37.898095969 (595h6m37.769179004s) [seg_pos={0 0}, upload=521677]: Done begin upload segment. Dec 30 10:35:37.898126226 (595h6m37.769209261s) [noise=true , address=145.79.227.28:4726 , piece_id=USLT7VCJXW3XIUBKJIWEZXDQW3UCNU7OBJE5W5FA656CV7Z7RBJQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.898191080 (595h6m37.769274115s) [noise=true , address=77.90.31.13:4626 , piece_id=M7X6D4CJSGVAKXBETA325RFNF7766XPWVVIBUWDTCKPXUW2KU6DQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.898892445 (595h6m37.76997548s) [noise=true , address=145.79.227.51:4409 , piece_id=CTBBMYJHQR6QKV234FMEWOXHOSDZFJCOBGIRBHTHQHCG63UO754A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.898908826 (595h6m37.769991861s) [noise=true , address=145.79.227.36:4519 , piece_id=274CAK6P7CEUKT6AZGS2IXVTPZVLVFDJ5ZSW6I5J5PCXZLVLCLAQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899241260 (595h6m37.770324305s) [noise=true , address=145.79.227.15:4873 , piece_id=V2LX57CMLFCHSI74REOYCTBYV2XASSNJ3MFSURWZHNDPH2GBLKYA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899249946 (595h6m37.770332981s) [noise=true , address=145.79.227.19:4079 , piece_id=NHL2UHODXNQOKXPVQMT3DDFGLE4ON3EJZCJW2HSXCVQKYM36XWDA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899499791 (595h6m37.770582826s) [noise=true , address=145.79.227.5:4221 , piece_id=A5U5FCSJB3JXSRALH2GJITQFWMKRP23PHDXI4D6FZMJPYTWGTALQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899570246 (595h6m37.770653281s) [noise=true , address=45.91.116.13:4153 , piece_id=3PAEOVMQUSNSDMHYIHZEIHKRUGCQF3XI3CXYWNYH4MQJX7W3UG3A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899758805 (595h6m37.77084184s) [noise=true , address=145.79.227.12:4765 , piece_id=HXLI3IRM2WLLXQZZZJDE7XEVDSMEXIAXJHPSDF3XI6AVE75E3SVA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899913429 (595h6m37.770996464s) [noise=true , address=145.79.227.22:4510 , piece_id=NPW6KWE3SIYWVUN4WSU5BO2BTEF3S5CYSTPDBBLG52Y6B4RF72QQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900031054 (595h6m37.771114089s) [noise=true , address=145.79.227.18:4006 , piece_id=MLZZKCQFU4ZBX4J5FUZHJ26CWSOMSJJHOSDQLGB37GWVHYHXNDUA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900277222 (595h6m37.771360258s) [noise=true , address=45.91.116.4:4937 , piece_id=IALTP3UKI4PULI2VVKPPTYY5Z2CGWZW3C5V3TYJC3I7G7LUZLM5A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900348268 (595h6m37.771431313s) [noise=true , address=145.79.227.34:4905 , piece_id=DFIORHRM57H5XRZGVIQYL3V4E4AGCWM7C7X2P7QZTWNW6GVQZVLQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900602161 (595h6m37.771685206s) [noise=true , address=145.79.227.7:4295 , piece_id=4RQS6473BFZSNAJD2VVEJNY2D52CVTLZKQ4MBEEJM53AUOL5TRSQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900657366 (595h6m37.771740401s) [noise=true , address=145.79.227.40:4050 , piece_id=YNGUQV7LIGRE2UZPSZXWHTAMZJE4GY2Y6WRKNHJ2LZWDWKEKLARQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.901035526 (595h6m37.772118571s) [noise=true , address=45.91.116.7:4012 , piece_id=T72JNTV4TUB23CC5XRPYWJZ5UCRE77WJHGAYM4SK2BAFWTDADY2A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:39.604350149 (595h6m39.475433194s) [upload=521677]: Next returned nil segment Dec 30 10:35:39.604382581 (595h6m39.475465616s) [upload=521677]: Waiting for error group managing segments... Dec 30 10:35:39.877515232 (595h6m39.748598267s) [noise=true , address=145.79.227.12:4765 , piece_id=HXLI3IRM2WLLXQZZZJDE7XEVDSMEXIAXJHPSDF3XI6AVE75E3SVA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.882503961 (595h6m39.753586996s) [noise=true , address=145.79.227.15:4873 , piece_id=V2LX57CMLFCHSI74REOYCTBYV2XASSNJ3MFSURWZHNDPH2GBLKYA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.888169319 (595h6m39.759252355s) [noise=true , address=145.79.227.19:4079 , piece_id=NHL2UHODXNQOKXPVQMT3DDFGLE4ON3EJZCJW2HSXCVQKYM36XWDA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.888523183 (595h6m39.759606228s) [noise=true , address=145.79.227.22:4510 , piece_id=NPW6KWE3SIYWVUN4WSU5BO2BTEF3S5CYSTPDBBLG52Y6B4RF72QQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.901555459 (595h6m39.772638505s) [noise=true , address=145.79.227.36:4519 , piece_id=274CAK6P7CEUKT6AZGS2IXVTPZVLVFDJ5ZSW6I5J5PCXZLVLCLAQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.904746936 (595h6m39.775829981s) [noise=true , address=145.79.227.51:4409 , piece_id=CTBBMYJHQR6QKV234FMEWOXHOSDZFJCOBGIRBHTHQHCG63UO754A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.910736492 (595h6m39.781819537s) [noise=true , address=145.79.227.7:4295 , piece_id=4RQS6473BFZSNAJD2VVEJNY2D52CVTLZKQ4MBEEJM53AUOL5TRSQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.974959302 (595h6m39.846042337s) [noise=true , address=145.79.227.5:4221 , piece_id=A5U5FCSJB3JXSRALH2GJITQFWMKRP23PHDXI4D6FZMJPYTWGTALQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.981452888 (595h6m39.852535923s) [noise=true , address=145.79.227.18:4006 , piece_id=MLZZKCQFU4ZBX4J5FUZHJ26CWSOMSJJHOSDQLGB37GWVHYHXNDUA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.982670747 (595h6m39.853753782s) [noise=true , address=145.79.227.28:4726 , piece_id=USLT7VCJXW3XIUBKJIWEZXDQW3UCNU7OBJE5W5FA656CV7Z7RBJQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.209859583 (595h6m40.080942618s) [noise=true , address=45.91.116.13:4153 , piece_id=3PAEOVMQUSNSDMHYIHZEIHKRUGCQF3XI3CXYWNYH4MQJX7W3UG3A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.287338457 (595h6m40.158421492s) [noise=true , address=45.91.116.4:4937 , piece_id=IALTP3UKI4PULI2VVKPPTYY5Z2CGWZW3C5V3TYJC3I7G7LUZLM5A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.500418026 (595h6m40.371501071s) [noise=true , address=77.90.31.13:4626 , piece_id=M7X6D4CJSGVAKXBETA325RFNF7766XPWVVIBUWDTCKPXUW2KU6DQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.500440168 (595h6m40.371523203s) [seg_pos={0 0}, upload=521677]: Segment reached optimal threshold of 13 pieces. Dec 30 10:35:40.500702026 (595h6m40.371785092s) [noise=true , address=145.79.227.34:4905 , piece_id=DFIORHRM57H5XRZGVIQYL3V4E4AGCWM7C7X2P7QZTWNW6GVQZVLQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 30 10:35:40.500712136 (595h6m40.371795181s) [noise=true , address=145.79.227.34:4905 , piece_id=DFIORHRM57H5XRZGVIQYL3V4E4AGCWM7C7X2P7QZTWNW6GVQZVLQ, seg_pos={0 0}, upload=521677]: Long tail cancellation due to optimal threshold Dec 30 10:35:40.500795284 (595h6m40.371878329s) [noise=true , address=145.79.227.40:4050 , piece_id=YNGUQV7LIGRE2UZPSZXWHTAMZJE4GY2Y6WRKNHJ2LZWDWKEKLARQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 10:35:40.500824300 (595h6m40.371907345s) [noise=true , address=145.79.227.40:4050 , piece_id=YNGUQV7LIGRE2UZPSZXWHTAMZJE4GY2Y6WRKNHJ2LZWDWKEKLARQ, seg_pos={0 0}, upload=521677]: Long tail cancellation due to optimal threshold Dec 30 10:35:40.501039109 (595h6m40.372122144s) [noise=true , address=45.91.116.7:4012 , piece_id=T72JNTV4TUB23CC5XRPYWJZ5UCRE77WJHGAYM4SK2BAFWTDADY2A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 30 10:35:40.501047384 (595h6m40.372130419s) [noise=true , address=45.91.116.7:4012 , piece_id=T72JNTV4TUB23CC5XRPYWJZ5UCRE77WJHGAYM4SK2BAFWTDADY2A, seg_pos={0 0}, upload=521677]: Long tail cancellation due to optimal threshold Dec 30 10:35:40.501068395 (595h6m40.37215144s) [seg_pos={0 0}, upload=521677]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:35:40.501138688 (595h6m40.372221733s) [upload=521677]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:35:40.501143888 (595h6m40.372226933s) [upload=521677]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:35:40.800282569 (595h6m40.671365605s) [upload=521677]: Done uploading segments... Dec 30 10:35:40.800309010 (595h6m40.671392045s) [upload=521677]: Upload finished. err: <nil> Duration: 6.067847778s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................X....X.........X...............] duration: 40.136687324s trace: https://jaeger.dev.storj.io/trace/1dc8263b2e55d7dc FAIL (2025-12-30 10:40:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311300688 (594h58m28.182383733s) [noise=true , address=145.79.227.43:4179 , piece_id=CQ32O4X3PKQN37OWC3WU5Z5FF7CCAIOAWLUISOXJTTIXTSMKYNQA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311642950 (594h58m28.182725995s) [noise=true , address=145.79.227.30:4783 , piece_id=UIXZYYXWC7KJZUV3BNZAIUSZNDEYA5RFFLKMXWF3XLNKJF5K2A6Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311712022 (594h58m28.182795057s) [noise=true , address=145.79.227.35:4492 , piece_id=GM7TCP65VSFAGWDCRG4VKWAPJHU5HGZFZTVAD4XM6PJFMLCC4NRA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312038483 (594h58m28.183121528s) [noise=true , address=77.90.31.17:4798 , piece_id=HT7E2VVRR2IURW5OIXQ4BLR55QI2BEKJG3VIRG4CABVPCUZRIGWA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312100291 (594h58m28.183183326s) [noise=true , address=77.90.31.19:4859 , piece_id=HEZMENV33Y6A6MMMMDIUECW44662O24WTIG2RWCWZL3IGNKWCH2A, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312494201 (594h58m28.183577246s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312500252 (594h58m28.183583287s) [noise=true , address=145.79.227.49:4359 , piece_id=RGCUJ32JS6NXVVLH7HWUD5IZNV3HZPRRVIHEBJEII2JPEENVIKZA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312899192 (594h58m28.183982227s) [noise=true , address=145.79.227.18:4037 , piece_id=IQZILX3EVK57WA5QGC4VGRYTYWQX37PLIASYIYXIHWBWKGTSEESA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313265008 (594h58m28.184348053s) [noise=true , address=145.79.227.6:4267 , piece_id=ZE5IVJYY2XEW3QOZBBE7LYICAALYG3DZHU6K7UDYUGBHRTWPQYPA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313640203 (594h58m28.184723248s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313904957 (594h58m28.184988012s) [noise=true , address=45.91.116.14:4213 , piece_id=BUTDQ3PI56Q7ZHNYHCEMWPHJUI42LXBR42ZGF4VKMDXJ47BSPU3Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314038000 (594h58m28.185121045s) [noise=true , address=77.90.31.12:4603 , piece_id=QQ5V6RXCHOHVL2QTWLLRFPJBAQQ5KW47T2N44EBMIUW747Q4ZOAA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314410920 (594h58m28.185493965s) [noise=true , address=145.79.227.44:4219 , piece_id=NSHMZDOI264JBC5ML4GRTH66FKT76C5QQ534UA4QDFEIG5P6CHDQ, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314664292 (594h58m28.185747338s) [noise=true , address=145.79.227.40:4062 , piece_id=RTSWJJIEADDCF5Z4JMT4NERSF7EXCWP7QZKDN4KIWEX54UDLNZEA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.315164305 (594h58m28.18624734s) [noise=true , address=77.90.31.10:4540 , piece_id=5LXEPFE5MNCGS6KJK4RP5Q5MV4M3TQMVR5FK4DYTYY4MKTFXTP2Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:29.778852769 (594h58m29.649935804s) [upload=521557]: Next returned nil segment Dec 30 10:27:29.778904818 (594h58m29.649987863s) [upload=521557]: Waiting for error group managing segments... Dec 30 10:27:30.177976438 (594h58m30.049059474s) [noise=true , address=145.79.227.18:4037 , piece_id=IQZILX3EVK57WA5QGC4VGRYTYWQX37PLIASYIYXIHWBWKGTSEESA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.239401413 (594h58m30.110484448s) [noise=true , address=145.79.227.6:4267 , piece_id=ZE5IVJYY2XEW3QOZBBE7LYICAALYG3DZHU6K7UDYUGBHRTWPQYPA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.247231673 (594h58m30.118314708s) [noise=true , address=145.79.227.35:4492 , piece_id=GM7TCP65VSFAGWDCRG4VKWAPJHU5HGZFZTVAD4XM6PJFMLCC4NRA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.364460186 (594h58m30.235543221s) [noise=true , address=77.90.31.17:4798 , piece_id=HT7E2VVRR2IURW5OIXQ4BLR55QI2BEKJG3VIRG4CABVPCUZRIGWA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.415614833 (594h58m30.286697868s) [noise=true , address=77.90.31.12:4603 , piece_id=QQ5V6RXCHOHVL2QTWLLRFPJBAQQ5KW47T2N44EBMIUW747Q4ZOAA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.444886882 (594h58m30.315969917s) [noise=true , address=77.90.31.10:4540 , piece_id=5LXEPFE5MNCGS6KJK4RP5Q5MV4M3TQMVR5FK4DYTYY4MKTFXTP2Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.512047427 (594h58m30.383130462s) [noise=true , address=45.91.116.14:4213 , piece_id=BUTDQ3PI56Q7ZHNYHCEMWPHJUI42LXBR42ZGF4VKMDXJ47BSPU3Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.536847771 (594h58m30.407930806s) [noise=true , address=77.90.31.19:4859 , piece_id=HEZMENV33Y6A6MMMMDIUECW44662O24WTIG2RWCWZL3IGNKWCH2A, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.680580563 (594h58m30.551663608s) [noise=true , address=45.91.116.12:4253 , piece_id=NPLIO7NLEIRTQAWDQ7WZVZE5ZSRLDPDNM34M6SEI53G5XEHB3LJA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.065501378 (594h58m30.936584414s) [noise=true , address=145.79.227.49:4359 , piece_id=RGCUJ32JS6NXVVLH7HWUD5IZNV3HZPRRVIHEBJEII2JPEENVIKZA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.104555514 (594h58m30.975638559s) [noise=true , address=145.79.227.44:4219 , piece_id=NSHMZDOI264JBC5ML4GRTH66FKT76C5QQ534UA4QDFEIG5P6CHDQ, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.115841635 (594h58m30.98692467s) [noise=true , address=145.79.227.30:4783 , piece_id=UIXZYYXWC7KJZUV3BNZAIUSZNDEYA5RFFLKMXWF3XLNKJF5K2A6Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.364792710 (594h58m31.235875755s) [noise=true , address=145.79.227.40:4062 , piece_id=RTSWJJIEADDCF5Z4JMT4NERSF7EXCWP7QZKDN4KIWEX54UDLNZEA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.374552906 (594h58m31.245635941s) [noise=true , address=145.79.227.43:4179 , piece_id=CQ32O4X3PKQN37OWC3WU5Z5FF7CCAIOAWLUISOXJTTIXTSMKYNQA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.374572984 (594h58m31.245656019s) [seg_pos={0 0}, upload=521557]: Segment reached optimal threshold of 14 pieces. Dec 30 10:27:31.374775900 (594h58m31.245858935s) [noise=true , address=45.91.116.10:4115 , piece_id=DRZ2YOFTPSE7SQD6MJ6YQWRQN6MDOLHHP7LWJMKMCPHO37A72C4A, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1arcuTW5rwnvfiqi2H7dL3oL6jKBkDeGq6DmgRGpS9Vs9BCHmf): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.374782553 (594h58m31.245865598s) [noise=true , address=45.91.116.10:4115 , piece_id=DRZ2YOFTPSE7SQD6MJ6YQWRQN6MDOLHHP7LWJMKMCPHO37A72C4A, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.374972405 (594h58m31.24605544s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.374978286 (594h58m31.246061321s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375019294 (594h58m31.246102349s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.375038921 (594h58m31.246121966s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375091411 (594h58m31.246174446s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE): protocol: write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection; write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection Dec 30 10:27:31.375099606 (594h58m31.246182641s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375195900 (594h58m31.246278935s) [seg_pos={0 0}, upload=521557]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:27:31.375261705 (594h58m31.24634474s) [upload=521557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:27:31.375266604 (594h58m31.246349639s) [upload=521557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:27:31.664767884 (594h58m31.53585099s) [upload=521557]: Done uploading segments... Dec 30 10:27:31.664793884 (594h58m31.535876929s) [upload=521557]: Upload finished. err: <nil> Duration: 40.136709075s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [.X......X...X...X........X.....X.X............X......X.X.XXX] duration: 5.001397252s trace: https://jaeger.dev.storj.io/trace/7863865c74c0ac82 FAIL (2025-12-30 10:40:13 UTC) orbiter warning: context deadline exceeded Duration: 5.001494948s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...XX.X.XX.X.X.X.XX.X.....XX..........XX...X.X..XX.XX.......] duration: 5.825350304s trace: https://jaeger.dev.storj.io/trace/495530043a5329b3 FAIL (2025-12-30 10:40:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: M7X6D4CJSGVAKXBETA325RFNF7766XPWVVIBUWDTCKPXUW2KU6DQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: 3PAEOVMQUSNSDMHYIHZEIHKRUGCQF3XI3CXYWNYH4MQJX7W3UG3A) context deadline exceeded ======LOGS====== Dec 30 10:35:37.649698347 (595h6m37.520781382s) [upload=521677]: Starting upload Dec 30 10:35:37.649784342 (595h6m37.520867397s) [upload=521677]: Done starting upload Dec 30 10:35:37.649861779 (595h6m37.520944814s) [upload=521677]: Uploading segments... Dec 30 10:35:37.650077299 (595h6m37.521160334s) [upload=521677]: Got next segment. Inline: false Dec 30 10:35:37.650127594 (595h6m37.521210639s) [upload=521677]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:35:37.650133907 (595h6m37.521216952s) [upload=521677]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:35:37.898044742 (595h6m37.769127777s) [seg_pos={0 0}, upload=521677]: Begin upload segment... Dec 30 10:35:37.898095969 (595h6m37.769179004s) [seg_pos={0 0}, upload=521677]: Done begin upload segment. Dec 30 10:35:37.898126226 (595h6m37.769209261s) [noise=true , address=145.79.227.28:4726 , piece_id=USLT7VCJXW3XIUBKJIWEZXDQW3UCNU7OBJE5W5FA656CV7Z7RBJQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.898191080 (595h6m37.769274115s) [noise=true , address=77.90.31.13:4626 , piece_id=M7X6D4CJSGVAKXBETA325RFNF7766XPWVVIBUWDTCKPXUW2KU6DQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.898892445 (595h6m37.76997548s) [noise=true , address=145.79.227.51:4409 , piece_id=CTBBMYJHQR6QKV234FMEWOXHOSDZFJCOBGIRBHTHQHCG63UO754A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.898908826 (595h6m37.769991861s) [noise=true , address=145.79.227.36:4519 , piece_id=274CAK6P7CEUKT6AZGS2IXVTPZVLVFDJ5ZSW6I5J5PCXZLVLCLAQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899241260 (595h6m37.770324305s) [noise=true , address=145.79.227.15:4873 , piece_id=V2LX57CMLFCHSI74REOYCTBYV2XASSNJ3MFSURWZHNDPH2GBLKYA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899249946 (595h6m37.770332981s) [noise=true , address=145.79.227.19:4079 , piece_id=NHL2UHODXNQOKXPVQMT3DDFGLE4ON3EJZCJW2HSXCVQKYM36XWDA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899499791 (595h6m37.770582826s) [noise=true , address=145.79.227.5:4221 , piece_id=A5U5FCSJB3JXSRALH2GJITQFWMKRP23PHDXI4D6FZMJPYTWGTALQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899570246 (595h6m37.770653281s) [noise=true , address=45.91.116.13:4153 , piece_id=3PAEOVMQUSNSDMHYIHZEIHKRUGCQF3XI3CXYWNYH4MQJX7W3UG3A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899758805 (595h6m37.77084184s) [noise=true , address=145.79.227.12:4765 , piece_id=HXLI3IRM2WLLXQZZZJDE7XEVDSMEXIAXJHPSDF3XI6AVE75E3SVA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899913429 (595h6m37.770996464s) [noise=true , address=145.79.227.22:4510 , piece_id=NPW6KWE3SIYWVUN4WSU5BO2BTEF3S5CYSTPDBBLG52Y6B4RF72QQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900031054 (595h6m37.771114089s) [noise=true , address=145.79.227.18:4006 , piece_id=MLZZKCQFU4ZBX4J5FUZHJ26CWSOMSJJHOSDQLGB37GWVHYHXNDUA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900277222 (595h6m37.771360258s) [noise=true , address=45.91.116.4:4937 , piece_id=IALTP3UKI4PULI2VVKPPTYY5Z2CGWZW3C5V3TYJC3I7G7LUZLM5A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900348268 (595h6m37.771431313s) [noise=true , address=145.79.227.34:4905 , piece_id=DFIORHRM57H5XRZGVIQYL3V4E4AGCWM7C7X2P7QZTWNW6GVQZVLQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900602161 (595h6m37.771685206s) [noise=true , address=145.79.227.7:4295 , piece_id=4RQS6473BFZSNAJD2VVEJNY2D52CVTLZKQ4MBEEJM53AUOL5TRSQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900657366 (595h6m37.771740401s) [noise=true , address=145.79.227.40:4050 , piece_id=YNGUQV7LIGRE2UZPSZXWHTAMZJE4GY2Y6WRKNHJ2LZWDWKEKLARQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.901035526 (595h6m37.772118571s) [noise=true , address=45.91.116.7:4012 , piece_id=T72JNTV4TUB23CC5XRPYWJZ5UCRE77WJHGAYM4SK2BAFWTDADY2A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:39.604350149 (595h6m39.475433194s) [upload=521677]: Next returned nil segment Dec 30 10:35:39.604382581 (595h6m39.475465616s) [upload=521677]: Waiting for error group managing segments... Dec 30 10:35:39.877515232 (595h6m39.748598267s) [noise=true , address=145.79.227.12:4765 , piece_id=HXLI3IRM2WLLXQZZZJDE7XEVDSMEXIAXJHPSDF3XI6AVE75E3SVA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.882503961 (595h6m39.753586996s) [noise=true , address=145.79.227.15:4873 , piece_id=V2LX57CMLFCHSI74REOYCTBYV2XASSNJ3MFSURWZHNDPH2GBLKYA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.888169319 (595h6m39.759252355s) [noise=true , address=145.79.227.19:4079 , piece_id=NHL2UHODXNQOKXPVQMT3DDFGLE4ON3EJZCJW2HSXCVQKYM36XWDA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.888523183 (595h6m39.759606228s) [noise=true , address=145.79.227.22:4510 , piece_id=NPW6KWE3SIYWVUN4WSU5BO2BTEF3S5CYSTPDBBLG52Y6B4RF72QQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.901555459 (595h6m39.772638505s) [noise=true , address=145.79.227.36:4519 , piece_id=274CAK6P7CEUKT6AZGS2IXVTPZVLVFDJ5ZSW6I5J5PCXZLVLCLAQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.904746936 (595h6m39.775829981s) [noise=true , address=145.79.227.51:4409 , piece_id=CTBBMYJHQR6QKV234FMEWOXHOSDZFJCOBGIRBHTHQHCG63UO754A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.910736492 (595h6m39.781819537s) [noise=true , address=145.79.227.7:4295 , piece_id=4RQS6473BFZSNAJD2VVEJNY2D52CVTLZKQ4MBEEJM53AUOL5TRSQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.974959302 (595h6m39.846042337s) [noise=true , address=145.79.227.5:4221 , piece_id=A5U5FCSJB3JXSRALH2GJITQFWMKRP23PHDXI4D6FZMJPYTWGTALQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.981452888 (595h6m39.852535923s) [noise=true , address=145.79.227.18:4006 , piece_id=MLZZKCQFU4ZBX4J5FUZHJ26CWSOMSJJHOSDQLGB37GWVHYHXNDUA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.982670747 (595h6m39.853753782s) [noise=true , address=145.79.227.28:4726 , piece_id=USLT7VCJXW3XIUBKJIWEZXDQW3UCNU7OBJE5W5FA656CV7Z7RBJQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.209859583 (595h6m40.080942618s) [noise=true , address=45.91.116.13:4153 , piece_id=3PAEOVMQUSNSDMHYIHZEIHKRUGCQF3XI3CXYWNYH4MQJX7W3UG3A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.287338457 (595h6m40.158421492s) [noise=true , address=45.91.116.4:4937 , piece_id=IALTP3UKI4PULI2VVKPPTYY5Z2CGWZW3C5V3TYJC3I7G7LUZLM5A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.500418026 (595h6m40.371501071s) [noise=true , address=77.90.31.13:4626 , piece_id=M7X6D4CJSGVAKXBETA325RFNF7766XPWVVIBUWDTCKPXUW2KU6DQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.500440168 (595h6m40.371523203s) [seg_pos={0 0}, upload=521677]: Segment reached optimal threshold of 13 pieces. Dec 30 10:35:40.500702026 (595h6m40.371785092s) [noise=true , address=145.79.227.34:4905 , piece_id=DFIORHRM57H5XRZGVIQYL3V4E4AGCWM7C7X2P7QZTWNW6GVQZVLQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 30 10:35:40.500712136 (595h6m40.371795181s) [noise=true , address=145.79.227.34:4905 , piece_id=DFIORHRM57H5XRZGVIQYL3V4E4AGCWM7C7X2P7QZTWNW6GVQZVLQ, seg_pos={0 0}, upload=521677]: Long tail cancellation due to optimal threshold Dec 30 10:35:40.500795284 (595h6m40.371878329s) [noise=true , address=145.79.227.40:4050 , piece_id=YNGUQV7LIGRE2UZPSZXWHTAMZJE4GY2Y6WRKNHJ2LZWDWKEKLARQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 10:35:40.500824300 (595h6m40.371907345s) [noise=true , address=145.79.227.40:4050 , piece_id=YNGUQV7LIGRE2UZPSZXWHTAMZJE4GY2Y6WRKNHJ2LZWDWKEKLARQ, seg_pos={0 0}, upload=521677]: Long tail cancellation due to optimal threshold Dec 30 10:35:40.501039109 (595h6m40.372122144s) [noise=true , address=45.91.116.7:4012 , piece_id=T72JNTV4TUB23CC5XRPYWJZ5UCRE77WJHGAYM4SK2BAFWTDADY2A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 30 10:35:40.501047384 (595h6m40.372130419s) [noise=true , address=45.91.116.7:4012 , piece_id=T72JNTV4TUB23CC5XRPYWJZ5UCRE77WJHGAYM4SK2BAFWTDADY2A, seg_pos={0 0}, upload=521677]: Long tail cancellation due to optimal threshold Dec 30 10:35:40.501068395 (595h6m40.37215144s) [seg_pos={0 0}, upload=521677]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:35:40.501138688 (595h6m40.372221733s) [upload=521677]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:35:40.501143888 (595h6m40.372226933s) [upload=521677]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:35:40.800282569 (595h6m40.671365605s) [upload=521677]: Done uploading segments... Dec 30 10:35:40.800309010 (595h6m40.671392045s) [upload=521677]: Upload finished. err: <nil> Duration: 5.825419425s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [....XX.X.XX.X.X.X.XX.X.....XX..........XX...X.X..XX.XX......] duration: 5.702280617s trace: https://jaeger.dev.storj.io/trace/495530043a5329b3 FAIL (2025-12-30 10:39:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: M7X6D4CJSGVAKXBETA325RFNF7766XPWVVIBUWDTCKPXUW2KU6DQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: 3PAEOVMQUSNSDMHYIHZEIHKRUGCQF3XI3CXYWNYH4MQJX7W3UG3A) context deadline exceeded ======LOGS====== Dec 30 10:35:37.649698347 (595h6m37.520781382s) [upload=521677]: Starting upload Dec 30 10:35:37.649784342 (595h6m37.520867397s) [upload=521677]: Done starting upload Dec 30 10:35:37.649861779 (595h6m37.520944814s) [upload=521677]: Uploading segments... Dec 30 10:35:37.650077299 (595h6m37.521160334s) [upload=521677]: Got next segment. Inline: false Dec 30 10:35:37.650127594 (595h6m37.521210639s) [upload=521677]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:35:37.650133907 (595h6m37.521216952s) [upload=521677]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:35:37.898044742 (595h6m37.769127777s) [seg_pos={0 0}, upload=521677]: Begin upload segment... Dec 30 10:35:37.898095969 (595h6m37.769179004s) [seg_pos={0 0}, upload=521677]: Done begin upload segment. Dec 30 10:35:37.898126226 (595h6m37.769209261s) [noise=true , address=145.79.227.28:4726 , piece_id=USLT7VCJXW3XIUBKJIWEZXDQW3UCNU7OBJE5W5FA656CV7Z7RBJQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.898191080 (595h6m37.769274115s) [noise=true , address=77.90.31.13:4626 , piece_id=M7X6D4CJSGVAKXBETA325RFNF7766XPWVVIBUWDTCKPXUW2KU6DQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.898892445 (595h6m37.76997548s) [noise=true , address=145.79.227.51:4409 , piece_id=CTBBMYJHQR6QKV234FMEWOXHOSDZFJCOBGIRBHTHQHCG63UO754A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.898908826 (595h6m37.769991861s) [noise=true , address=145.79.227.36:4519 , piece_id=274CAK6P7CEUKT6AZGS2IXVTPZVLVFDJ5ZSW6I5J5PCXZLVLCLAQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899241260 (595h6m37.770324305s) [noise=true , address=145.79.227.15:4873 , piece_id=V2LX57CMLFCHSI74REOYCTBYV2XASSNJ3MFSURWZHNDPH2GBLKYA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899249946 (595h6m37.770332981s) [noise=true , address=145.79.227.19:4079 , piece_id=NHL2UHODXNQOKXPVQMT3DDFGLE4ON3EJZCJW2HSXCVQKYM36XWDA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899499791 (595h6m37.770582826s) [noise=true , address=145.79.227.5:4221 , piece_id=A5U5FCSJB3JXSRALH2GJITQFWMKRP23PHDXI4D6FZMJPYTWGTALQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899570246 (595h6m37.770653281s) [noise=true , address=45.91.116.13:4153 , piece_id=3PAEOVMQUSNSDMHYIHZEIHKRUGCQF3XI3CXYWNYH4MQJX7W3UG3A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899758805 (595h6m37.77084184s) [noise=true , address=145.79.227.12:4765 , piece_id=HXLI3IRM2WLLXQZZZJDE7XEVDSMEXIAXJHPSDF3XI6AVE75E3SVA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899913429 (595h6m37.770996464s) [noise=true , address=145.79.227.22:4510 , piece_id=NPW6KWE3SIYWVUN4WSU5BO2BTEF3S5CYSTPDBBLG52Y6B4RF72QQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900031054 (595h6m37.771114089s) [noise=true , address=145.79.227.18:4006 , piece_id=MLZZKCQFU4ZBX4J5FUZHJ26CWSOMSJJHOSDQLGB37GWVHYHXNDUA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900277222 (595h6m37.771360258s) [noise=true , address=45.91.116.4:4937 , piece_id=IALTP3UKI4PULI2VVKPPTYY5Z2CGWZW3C5V3TYJC3I7G7LUZLM5A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900348268 (595h6m37.771431313s) [noise=true , address=145.79.227.34:4905 , piece_id=DFIORHRM57H5XRZGVIQYL3V4E4AGCWM7C7X2P7QZTWNW6GVQZVLQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900602161 (595h6m37.771685206s) [noise=true , address=145.79.227.7:4295 , piece_id=4RQS6473BFZSNAJD2VVEJNY2D52CVTLZKQ4MBEEJM53AUOL5TRSQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900657366 (595h6m37.771740401s) [noise=true , address=145.79.227.40:4050 , piece_id=YNGUQV7LIGRE2UZPSZXWHTAMZJE4GY2Y6WRKNHJ2LZWDWKEKLARQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.901035526 (595h6m37.772118571s) [noise=true , address=45.91.116.7:4012 , piece_id=T72JNTV4TUB23CC5XRPYWJZ5UCRE77WJHGAYM4SK2BAFWTDADY2A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:39.604350149 (595h6m39.475433194s) [upload=521677]: Next returned nil segment Dec 30 10:35:39.604382581 (595h6m39.475465616s) [upload=521677]: Waiting for error group managing segments... Dec 30 10:35:39.877515232 (595h6m39.748598267s) [noise=true , address=145.79.227.12:4765 , piece_id=HXLI3IRM2WLLXQZZZJDE7XEVDSMEXIAXJHPSDF3XI6AVE75E3SVA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.882503961 (595h6m39.753586996s) [noise=true , address=145.79.227.15:4873 , piece_id=V2LX57CMLFCHSI74REOYCTBYV2XASSNJ3MFSURWZHNDPH2GBLKYA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.888169319 (595h6m39.759252355s) [noise=true , address=145.79.227.19:4079 , piece_id=NHL2UHODXNQOKXPVQMT3DDFGLE4ON3EJZCJW2HSXCVQKYM36XWDA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.888523183 (595h6m39.759606228s) [noise=true , address=145.79.227.22:4510 , piece_id=NPW6KWE3SIYWVUN4WSU5BO2BTEF3S5CYSTPDBBLG52Y6B4RF72QQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.901555459 (595h6m39.772638505s) [noise=true , address=145.79.227.36:4519 , piece_id=274CAK6P7CEUKT6AZGS2IXVTPZVLVFDJ5ZSW6I5J5PCXZLVLCLAQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.904746936 (595h6m39.775829981s) [noise=true , address=145.79.227.51:4409 , piece_id=CTBBMYJHQR6QKV234FMEWOXHOSDZFJCOBGIRBHTHQHCG63UO754A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.910736492 (595h6m39.781819537s) [noise=true , address=145.79.227.7:4295 , piece_id=4RQS6473BFZSNAJD2VVEJNY2D52CVTLZKQ4MBEEJM53AUOL5TRSQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.974959302 (595h6m39.846042337s) [noise=true , address=145.79.227.5:4221 , piece_id=A5U5FCSJB3JXSRALH2GJITQFWMKRP23PHDXI4D6FZMJPYTWGTALQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.981452888 (595h6m39.852535923s) [noise=true , address=145.79.227.18:4006 , piece_id=MLZZKCQFU4ZBX4J5FUZHJ26CWSOMSJJHOSDQLGB37GWVHYHXNDUA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.982670747 (595h6m39.853753782s) [noise=true , address=145.79.227.28:4726 , piece_id=USLT7VCJXW3XIUBKJIWEZXDQW3UCNU7OBJE5W5FA656CV7Z7RBJQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.209859583 (595h6m40.080942618s) [noise=true , address=45.91.116.13:4153 , piece_id=3PAEOVMQUSNSDMHYIHZEIHKRUGCQF3XI3CXYWNYH4MQJX7W3UG3A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.287338457 (595h6m40.158421492s) [noise=true , address=45.91.116.4:4937 , piece_id=IALTP3UKI4PULI2VVKPPTYY5Z2CGWZW3C5V3TYJC3I7G7LUZLM5A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.500418026 (595h6m40.371501071s) [noise=true , address=77.90.31.13:4626 , piece_id=M7X6D4CJSGVAKXBETA325RFNF7766XPWVVIBUWDTCKPXUW2KU6DQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.500440168 (595h6m40.371523203s) [seg_pos={0 0}, upload=521677]: Segment reached optimal threshold of 13 pieces. Dec 30 10:35:40.500702026 (595h6m40.371785092s) [noise=true , address=145.79.227.34:4905 , piece_id=DFIORHRM57H5XRZGVIQYL3V4E4AGCWM7C7X2P7QZTWNW6GVQZVLQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 30 10:35:40.500712136 (595h6m40.371795181s) [noise=true , address=145.79.227.34:4905 , piece_id=DFIORHRM57H5XRZGVIQYL3V4E4AGCWM7C7X2P7QZTWNW6GVQZVLQ, seg_pos={0 0}, upload=521677]: Long tail cancellation due to optimal threshold Dec 30 10:35:40.500795284 (595h6m40.371878329s) [noise=true , address=145.79.227.40:4050 , piece_id=YNGUQV7LIGRE2UZPSZXWHTAMZJE4GY2Y6WRKNHJ2LZWDWKEKLARQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 10:35:40.500824300 (595h6m40.371907345s) [noise=true , address=145.79.227.40:4050 , piece_id=YNGUQV7LIGRE2UZPSZXWHTAMZJE4GY2Y6WRKNHJ2LZWDWKEKLARQ, seg_pos={0 0}, upload=521677]: Long tail cancellation due to optimal threshold Dec 30 10:35:40.501039109 (595h6m40.372122144s) [noise=true , address=45.91.116.7:4012 , piece_id=T72JNTV4TUB23CC5XRPYWJZ5UCRE77WJHGAYM4SK2BAFWTDADY2A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 30 10:35:40.501047384 (595h6m40.372130419s) [noise=true , address=45.91.116.7:4012 , piece_id=T72JNTV4TUB23CC5XRPYWJZ5UCRE77WJHGAYM4SK2BAFWTDADY2A, seg_pos={0 0}, upload=521677]: Long tail cancellation due to optimal threshold Dec 30 10:35:40.501068395 (595h6m40.37215144s) [seg_pos={0 0}, upload=521677]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:35:40.501138688 (595h6m40.372221733s) [upload=521677]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:35:40.501143888 (595h6m40.372226933s) [upload=521677]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:35:40.800282569 (595h6m40.671365605s) [upload=521677]: Done uploading segments... Dec 30 10:35:40.800309010 (595h6m40.671392045s) [upload=521677]: Upload finished. err: <nil> Duration: 5.70230886s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [..X......X...X...X........X.....X.X............X......X.X.XX] duration: 5.001076566s trace: https://jaeger.dev.storj.io/trace/4d95339114e126f1 FAIL (2025-12-30 10:39:10 UTC) orbiter warning: context deadline exceeded Duration: 5.001104438s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.....XX.X.XX.X.X.X.XX.X.....XX..........XX...X.X..XX.XX.....] duration: 5.80660626s trace: https://jaeger.dev.storj.io/trace/495530043a5329b3 FAIL (2025-12-30 10:39:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: M7X6D4CJSGVAKXBETA325RFNF7766XPWVVIBUWDTCKPXUW2KU6DQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: 3PAEOVMQUSNSDMHYIHZEIHKRUGCQF3XI3CXYWNYH4MQJX7W3UG3A) context deadline exceeded ======LOGS====== Dec 30 10:35:37.649698347 (595h6m37.520781382s) [upload=521677]: Starting upload Dec 30 10:35:37.649784342 (595h6m37.520867397s) [upload=521677]: Done starting upload Dec 30 10:35:37.649861779 (595h6m37.520944814s) [upload=521677]: Uploading segments... Dec 30 10:35:37.650077299 (595h6m37.521160334s) [upload=521677]: Got next segment. Inline: false Dec 30 10:35:37.650127594 (595h6m37.521210639s) [upload=521677]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:35:37.650133907 (595h6m37.521216952s) [upload=521677]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:35:37.898044742 (595h6m37.769127777s) [seg_pos={0 0}, upload=521677]: Begin upload segment... Dec 30 10:35:37.898095969 (595h6m37.769179004s) [seg_pos={0 0}, upload=521677]: Done begin upload segment. Dec 30 10:35:37.898126226 (595h6m37.769209261s) [noise=true , address=145.79.227.28:4726 , piece_id=USLT7VCJXW3XIUBKJIWEZXDQW3UCNU7OBJE5W5FA656CV7Z7RBJQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.898191080 (595h6m37.769274115s) [noise=true , address=77.90.31.13:4626 , piece_id=M7X6D4CJSGVAKXBETA325RFNF7766XPWVVIBUWDTCKPXUW2KU6DQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.898892445 (595h6m37.76997548s) [noise=true , address=145.79.227.51:4409 , piece_id=CTBBMYJHQR6QKV234FMEWOXHOSDZFJCOBGIRBHTHQHCG63UO754A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.898908826 (595h6m37.769991861s) [noise=true , address=145.79.227.36:4519 , piece_id=274CAK6P7CEUKT6AZGS2IXVTPZVLVFDJ5ZSW6I5J5PCXZLVLCLAQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899241260 (595h6m37.770324305s) [noise=true , address=145.79.227.15:4873 , piece_id=V2LX57CMLFCHSI74REOYCTBYV2XASSNJ3MFSURWZHNDPH2GBLKYA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899249946 (595h6m37.770332981s) [noise=true , address=145.79.227.19:4079 , piece_id=NHL2UHODXNQOKXPVQMT3DDFGLE4ON3EJZCJW2HSXCVQKYM36XWDA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899499791 (595h6m37.770582826s) [noise=true , address=145.79.227.5:4221 , piece_id=A5U5FCSJB3JXSRALH2GJITQFWMKRP23PHDXI4D6FZMJPYTWGTALQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899570246 (595h6m37.770653281s) [noise=true , address=45.91.116.13:4153 , piece_id=3PAEOVMQUSNSDMHYIHZEIHKRUGCQF3XI3CXYWNYH4MQJX7W3UG3A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899758805 (595h6m37.77084184s) [noise=true , address=145.79.227.12:4765 , piece_id=HXLI3IRM2WLLXQZZZJDE7XEVDSMEXIAXJHPSDF3XI6AVE75E3SVA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899913429 (595h6m37.770996464s) [noise=true , address=145.79.227.22:4510 , piece_id=NPW6KWE3SIYWVUN4WSU5BO2BTEF3S5CYSTPDBBLG52Y6B4RF72QQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900031054 (595h6m37.771114089s) [noise=true , address=145.79.227.18:4006 , piece_id=MLZZKCQFU4ZBX4J5FUZHJ26CWSOMSJJHOSDQLGB37GWVHYHXNDUA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900277222 (595h6m37.771360258s) [noise=true , address=45.91.116.4:4937 , piece_id=IALTP3UKI4PULI2VVKPPTYY5Z2CGWZW3C5V3TYJC3I7G7LUZLM5A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900348268 (595h6m37.771431313s) [noise=true , address=145.79.227.34:4905 , piece_id=DFIORHRM57H5XRZGVIQYL3V4E4AGCWM7C7X2P7QZTWNW6GVQZVLQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900602161 (595h6m37.771685206s) [noise=true , address=145.79.227.7:4295 , piece_id=4RQS6473BFZSNAJD2VVEJNY2D52CVTLZKQ4MBEEJM53AUOL5TRSQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900657366 (595h6m37.771740401s) [noise=true , address=145.79.227.40:4050 , piece_id=YNGUQV7LIGRE2UZPSZXWHTAMZJE4GY2Y6WRKNHJ2LZWDWKEKLARQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.901035526 (595h6m37.772118571s) [noise=true , address=45.91.116.7:4012 , piece_id=T72JNTV4TUB23CC5XRPYWJZ5UCRE77WJHGAYM4SK2BAFWTDADY2A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:39.604350149 (595h6m39.475433194s) [upload=521677]: Next returned nil segment Dec 30 10:35:39.604382581 (595h6m39.475465616s) [upload=521677]: Waiting for error group managing segments... Dec 30 10:35:39.877515232 (595h6m39.748598267s) [noise=true , address=145.79.227.12:4765 , piece_id=HXLI3IRM2WLLXQZZZJDE7XEVDSMEXIAXJHPSDF3XI6AVE75E3SVA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.882503961 (595h6m39.753586996s) [noise=true , address=145.79.227.15:4873 , piece_id=V2LX57CMLFCHSI74REOYCTBYV2XASSNJ3MFSURWZHNDPH2GBLKYA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.888169319 (595h6m39.759252355s) [noise=true , address=145.79.227.19:4079 , piece_id=NHL2UHODXNQOKXPVQMT3DDFGLE4ON3EJZCJW2HSXCVQKYM36XWDA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.888523183 (595h6m39.759606228s) [noise=true , address=145.79.227.22:4510 , piece_id=NPW6KWE3SIYWVUN4WSU5BO2BTEF3S5CYSTPDBBLG52Y6B4RF72QQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.901555459 (595h6m39.772638505s) [noise=true , address=145.79.227.36:4519 , piece_id=274CAK6P7CEUKT6AZGS2IXVTPZVLVFDJ5ZSW6I5J5PCXZLVLCLAQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.904746936 (595h6m39.775829981s) [noise=true , address=145.79.227.51:4409 , piece_id=CTBBMYJHQR6QKV234FMEWOXHOSDZFJCOBGIRBHTHQHCG63UO754A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.910736492 (595h6m39.781819537s) [noise=true , address=145.79.227.7:4295 , piece_id=4RQS6473BFZSNAJD2VVEJNY2D52CVTLZKQ4MBEEJM53AUOL5TRSQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.974959302 (595h6m39.846042337s) [noise=true , address=145.79.227.5:4221 , piece_id=A5U5FCSJB3JXSRALH2GJITQFWMKRP23PHDXI4D6FZMJPYTWGTALQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.981452888 (595h6m39.852535923s) [noise=true , address=145.79.227.18:4006 , piece_id=MLZZKCQFU4ZBX4J5FUZHJ26CWSOMSJJHOSDQLGB37GWVHYHXNDUA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.982670747 (595h6m39.853753782s) [noise=true , address=145.79.227.28:4726 , piece_id=USLT7VCJXW3XIUBKJIWEZXDQW3UCNU7OBJE5W5FA656CV7Z7RBJQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.209859583 (595h6m40.080942618s) [noise=true , address=45.91.116.13:4153 , piece_id=3PAEOVMQUSNSDMHYIHZEIHKRUGCQF3XI3CXYWNYH4MQJX7W3UG3A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.287338457 (595h6m40.158421492s) [noise=true , address=45.91.116.4:4937 , piece_id=IALTP3UKI4PULI2VVKPPTYY5Z2CGWZW3C5V3TYJC3I7G7LUZLM5A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.500418026 (595h6m40.371501071s) [noise=true , address=77.90.31.13:4626 , piece_id=M7X6D4CJSGVAKXBETA325RFNF7766XPWVVIBUWDTCKPXUW2KU6DQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.500440168 (595h6m40.371523203s) [seg_pos={0 0}, upload=521677]: Segment reached optimal threshold of 13 pieces. Dec 30 10:35:40.500702026 (595h6m40.371785092s) [noise=true , address=145.79.227.34:4905 , piece_id=DFIORHRM57H5XRZGVIQYL3V4E4AGCWM7C7X2P7QZTWNW6GVQZVLQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 30 10:35:40.500712136 (595h6m40.371795181s) [noise=true , address=145.79.227.34:4905 , piece_id=DFIORHRM57H5XRZGVIQYL3V4E4AGCWM7C7X2P7QZTWNW6GVQZVLQ, seg_pos={0 0}, upload=521677]: Long tail cancellation due to optimal threshold Dec 30 10:35:40.500795284 (595h6m40.371878329s) [noise=true , address=145.79.227.40:4050 , piece_id=YNGUQV7LIGRE2UZPSZXWHTAMZJE4GY2Y6WRKNHJ2LZWDWKEKLARQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 10:35:40.500824300 (595h6m40.371907345s) [noise=true , address=145.79.227.40:4050 , piece_id=YNGUQV7LIGRE2UZPSZXWHTAMZJE4GY2Y6WRKNHJ2LZWDWKEKLARQ, seg_pos={0 0}, upload=521677]: Long tail cancellation due to optimal threshold Dec 30 10:35:40.501039109 (595h6m40.372122144s) [noise=true , address=45.91.116.7:4012 , piece_id=T72JNTV4TUB23CC5XRPYWJZ5UCRE77WJHGAYM4SK2BAFWTDADY2A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 30 10:35:40.501047384 (595h6m40.372130419s) [noise=true , address=45.91.116.7:4012 , piece_id=T72JNTV4TUB23CC5XRPYWJZ5UCRE77WJHGAYM4SK2BAFWTDADY2A, seg_pos={0 0}, upload=521677]: Long tail cancellation due to optimal threshold Dec 30 10:35:40.501068395 (595h6m40.37215144s) [seg_pos={0 0}, upload=521677]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:35:40.501138688 (595h6m40.372221733s) [upload=521677]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:35:40.501143888 (595h6m40.372226933s) [upload=521677]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:35:40.800282569 (595h6m40.671365605s) [upload=521677]: Done uploading segments... Dec 30 10:35:40.800309010 (595h6m40.671392045s) [upload=521677]: Upload finished. err: <nil> Duration: 5.80663757s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................X....X.........X..............] duration: 16.269669332s trace: https://jaeger.dev.storj.io/trace/1dc8263b2e55d7dc FAIL (2025-12-30 10:38:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311300688 (594h58m28.182383733s) [noise=true , address=145.79.227.43:4179 , piece_id=CQ32O4X3PKQN37OWC3WU5Z5FF7CCAIOAWLUISOXJTTIXTSMKYNQA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311642950 (594h58m28.182725995s) [noise=true , address=145.79.227.30:4783 , piece_id=UIXZYYXWC7KJZUV3BNZAIUSZNDEYA5RFFLKMXWF3XLNKJF5K2A6Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311712022 (594h58m28.182795057s) [noise=true , address=145.79.227.35:4492 , piece_id=GM7TCP65VSFAGWDCRG4VKWAPJHU5HGZFZTVAD4XM6PJFMLCC4NRA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312038483 (594h58m28.183121528s) [noise=true , address=77.90.31.17:4798 , piece_id=HT7E2VVRR2IURW5OIXQ4BLR55QI2BEKJG3VIRG4CABVPCUZRIGWA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312100291 (594h58m28.183183326s) [noise=true , address=77.90.31.19:4859 , piece_id=HEZMENV33Y6A6MMMMDIUECW44662O24WTIG2RWCWZL3IGNKWCH2A, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312494201 (594h58m28.183577246s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312500252 (594h58m28.183583287s) [noise=true , address=145.79.227.49:4359 , piece_id=RGCUJ32JS6NXVVLH7HWUD5IZNV3HZPRRVIHEBJEII2JPEENVIKZA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312899192 (594h58m28.183982227s) [noise=true , address=145.79.227.18:4037 , piece_id=IQZILX3EVK57WA5QGC4VGRYTYWQX37PLIASYIYXIHWBWKGTSEESA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313265008 (594h58m28.184348053s) [noise=true , address=145.79.227.6:4267 , piece_id=ZE5IVJYY2XEW3QOZBBE7LYICAALYG3DZHU6K7UDYUGBHRTWPQYPA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313640203 (594h58m28.184723248s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313904957 (594h58m28.184988012s) [noise=true , address=45.91.116.14:4213 , piece_id=BUTDQ3PI56Q7ZHNYHCEMWPHJUI42LXBR42ZGF4VKMDXJ47BSPU3Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314038000 (594h58m28.185121045s) [noise=true , address=77.90.31.12:4603 , piece_id=QQ5V6RXCHOHVL2QTWLLRFPJBAQQ5KW47T2N44EBMIUW747Q4ZOAA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314410920 (594h58m28.185493965s) [noise=true , address=145.79.227.44:4219 , piece_id=NSHMZDOI264JBC5ML4GRTH66FKT76C5QQ534UA4QDFEIG5P6CHDQ, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314664292 (594h58m28.185747338s) [noise=true , address=145.79.227.40:4062 , piece_id=RTSWJJIEADDCF5Z4JMT4NERSF7EXCWP7QZKDN4KIWEX54UDLNZEA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.315164305 (594h58m28.18624734s) [noise=true , address=77.90.31.10:4540 , piece_id=5LXEPFE5MNCGS6KJK4RP5Q5MV4M3TQMVR5FK4DYTYY4MKTFXTP2Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:29.778852769 (594h58m29.649935804s) [upload=521557]: Next returned nil segment Dec 30 10:27:29.778904818 (594h58m29.649987863s) [upload=521557]: Waiting for error group managing segments... Dec 30 10:27:30.177976438 (594h58m30.049059474s) [noise=true , address=145.79.227.18:4037 , piece_id=IQZILX3EVK57WA5QGC4VGRYTYWQX37PLIASYIYXIHWBWKGTSEESA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.239401413 (594h58m30.110484448s) [noise=true , address=145.79.227.6:4267 , piece_id=ZE5IVJYY2XEW3QOZBBE7LYICAALYG3DZHU6K7UDYUGBHRTWPQYPA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.247231673 (594h58m30.118314708s) [noise=true , address=145.79.227.35:4492 , piece_id=GM7TCP65VSFAGWDCRG4VKWAPJHU5HGZFZTVAD4XM6PJFMLCC4NRA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.364460186 (594h58m30.235543221s) [noise=true , address=77.90.31.17:4798 , piece_id=HT7E2VVRR2IURW5OIXQ4BLR55QI2BEKJG3VIRG4CABVPCUZRIGWA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.415614833 (594h58m30.286697868s) [noise=true , address=77.90.31.12:4603 , piece_id=QQ5V6RXCHOHVL2QTWLLRFPJBAQQ5KW47T2N44EBMIUW747Q4ZOAA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.444886882 (594h58m30.315969917s) [noise=true , address=77.90.31.10:4540 , piece_id=5LXEPFE5MNCGS6KJK4RP5Q5MV4M3TQMVR5FK4DYTYY4MKTFXTP2Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.512047427 (594h58m30.383130462s) [noise=true , address=45.91.116.14:4213 , piece_id=BUTDQ3PI56Q7ZHNYHCEMWPHJUI42LXBR42ZGF4VKMDXJ47BSPU3Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.536847771 (594h58m30.407930806s) [noise=true , address=77.90.31.19:4859 , piece_id=HEZMENV33Y6A6MMMMDIUECW44662O24WTIG2RWCWZL3IGNKWCH2A, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.680580563 (594h58m30.551663608s) [noise=true , address=45.91.116.12:4253 , piece_id=NPLIO7NLEIRTQAWDQ7WZVZE5ZSRLDPDNM34M6SEI53G5XEHB3LJA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.065501378 (594h58m30.936584414s) [noise=true , address=145.79.227.49:4359 , piece_id=RGCUJ32JS6NXVVLH7HWUD5IZNV3HZPRRVIHEBJEII2JPEENVIKZA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.104555514 (594h58m30.975638559s) [noise=true , address=145.79.227.44:4219 , piece_id=NSHMZDOI264JBC5ML4GRTH66FKT76C5QQ534UA4QDFEIG5P6CHDQ, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.115841635 (594h58m30.98692467s) [noise=true , address=145.79.227.30:4783 , piece_id=UIXZYYXWC7KJZUV3BNZAIUSZNDEYA5RFFLKMXWF3XLNKJF5K2A6Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.364792710 (594h58m31.235875755s) [noise=true , address=145.79.227.40:4062 , piece_id=RTSWJJIEADDCF5Z4JMT4NERSF7EXCWP7QZKDN4KIWEX54UDLNZEA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.374552906 (594h58m31.245635941s) [noise=true , address=145.79.227.43:4179 , piece_id=CQ32O4X3PKQN37OWC3WU5Z5FF7CCAIOAWLUISOXJTTIXTSMKYNQA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.374572984 (594h58m31.245656019s) [seg_pos={0 0}, upload=521557]: Segment reached optimal threshold of 14 pieces. Dec 30 10:27:31.374775900 (594h58m31.245858935s) [noise=true , address=45.91.116.10:4115 , piece_id=DRZ2YOFTPSE7SQD6MJ6YQWRQN6MDOLHHP7LWJMKMCPHO37A72C4A, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1arcuTW5rwnvfiqi2H7dL3oL6jKBkDeGq6DmgRGpS9Vs9BCHmf): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.374782553 (594h58m31.245865598s) [noise=true , address=45.91.116.10:4115 , piece_id=DRZ2YOFTPSE7SQD6MJ6YQWRQN6MDOLHHP7LWJMKMCPHO37A72C4A, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.374972405 (594h58m31.24605544s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.374978286 (594h58m31.246061321s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375019294 (594h58m31.246102349s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.375038921 (594h58m31.246121966s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375091411 (594h58m31.246174446s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE): protocol: write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection; write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection Dec 30 10:27:31.375099606 (594h58m31.246182641s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375195900 (594h58m31.246278935s) [seg_pos={0 0}, upload=521557]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:27:31.375261705 (594h58m31.24634474s) [upload=521557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:27:31.375266604 (594h58m31.246349639s) [upload=521557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:27:31.664767884 (594h58m31.53585099s) [upload=521557]: Done uploading segments... Dec 30 10:27:31.664793884 (594h58m31.535876929s) [upload=521557]: Upload finished. err: <nil> Duration: 16.269696374s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................X....X.........X.............] duration: 36.498643807s trace: https://jaeger.dev.storj.io/trace/1dc8263b2e55d7dc FAIL (2025-12-30 10:38:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311300688 (594h58m28.182383733s) [noise=true , address=145.79.227.43:4179 , piece_id=CQ32O4X3PKQN37OWC3WU5Z5FF7CCAIOAWLUISOXJTTIXTSMKYNQA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311642950 (594h58m28.182725995s) [noise=true , address=145.79.227.30:4783 , piece_id=UIXZYYXWC7KJZUV3BNZAIUSZNDEYA5RFFLKMXWF3XLNKJF5K2A6Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311712022 (594h58m28.182795057s) [noise=true , address=145.79.227.35:4492 , piece_id=GM7TCP65VSFAGWDCRG4VKWAPJHU5HGZFZTVAD4XM6PJFMLCC4NRA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312038483 (594h58m28.183121528s) [noise=true , address=77.90.31.17:4798 , piece_id=HT7E2VVRR2IURW5OIXQ4BLR55QI2BEKJG3VIRG4CABVPCUZRIGWA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312100291 (594h58m28.183183326s) [noise=true , address=77.90.31.19:4859 , piece_id=HEZMENV33Y6A6MMMMDIUECW44662O24WTIG2RWCWZL3IGNKWCH2A, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312494201 (594h58m28.183577246s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312500252 (594h58m28.183583287s) [noise=true , address=145.79.227.49:4359 , piece_id=RGCUJ32JS6NXVVLH7HWUD5IZNV3HZPRRVIHEBJEII2JPEENVIKZA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312899192 (594h58m28.183982227s) [noise=true , address=145.79.227.18:4037 , piece_id=IQZILX3EVK57WA5QGC4VGRYTYWQX37PLIASYIYXIHWBWKGTSEESA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313265008 (594h58m28.184348053s) [noise=true , address=145.79.227.6:4267 , piece_id=ZE5IVJYY2XEW3QOZBBE7LYICAALYG3DZHU6K7UDYUGBHRTWPQYPA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313640203 (594h58m28.184723248s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313904957 (594h58m28.184988012s) [noise=true , address=45.91.116.14:4213 , piece_id=BUTDQ3PI56Q7ZHNYHCEMWPHJUI42LXBR42ZGF4VKMDXJ47BSPU3Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314038000 (594h58m28.185121045s) [noise=true , address=77.90.31.12:4603 , piece_id=QQ5V6RXCHOHVL2QTWLLRFPJBAQQ5KW47T2N44EBMIUW747Q4ZOAA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314410920 (594h58m28.185493965s) [noise=true , address=145.79.227.44:4219 , piece_id=NSHMZDOI264JBC5ML4GRTH66FKT76C5QQ534UA4QDFEIG5P6CHDQ, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314664292 (594h58m28.185747338s) [noise=true , address=145.79.227.40:4062 , piece_id=RTSWJJIEADDCF5Z4JMT4NERSF7EXCWP7QZKDN4KIWEX54UDLNZEA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.315164305 (594h58m28.18624734s) [noise=true , address=77.90.31.10:4540 , piece_id=5LXEPFE5MNCGS6KJK4RP5Q5MV4M3TQMVR5FK4DYTYY4MKTFXTP2Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:29.778852769 (594h58m29.649935804s) [upload=521557]: Next returned nil segment Dec 30 10:27:29.778904818 (594h58m29.649987863s) [upload=521557]: Waiting for error group managing segments... Dec 30 10:27:30.177976438 (594h58m30.049059474s) [noise=true , address=145.79.227.18:4037 , piece_id=IQZILX3EVK57WA5QGC4VGRYTYWQX37PLIASYIYXIHWBWKGTSEESA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.239401413 (594h58m30.110484448s) [noise=true , address=145.79.227.6:4267 , piece_id=ZE5IVJYY2XEW3QOZBBE7LYICAALYG3DZHU6K7UDYUGBHRTWPQYPA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.247231673 (594h58m30.118314708s) [noise=true , address=145.79.227.35:4492 , piece_id=GM7TCP65VSFAGWDCRG4VKWAPJHU5HGZFZTVAD4XM6PJFMLCC4NRA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.364460186 (594h58m30.235543221s) [noise=true , address=77.90.31.17:4798 , piece_id=HT7E2VVRR2IURW5OIXQ4BLR55QI2BEKJG3VIRG4CABVPCUZRIGWA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.415614833 (594h58m30.286697868s) [noise=true , address=77.90.31.12:4603 , piece_id=QQ5V6RXCHOHVL2QTWLLRFPJBAQQ5KW47T2N44EBMIUW747Q4ZOAA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.444886882 (594h58m30.315969917s) [noise=true , address=77.90.31.10:4540 , piece_id=5LXEPFE5MNCGS6KJK4RP5Q5MV4M3TQMVR5FK4DYTYY4MKTFXTP2Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.512047427 (594h58m30.383130462s) [noise=true , address=45.91.116.14:4213 , piece_id=BUTDQ3PI56Q7ZHNYHCEMWPHJUI42LXBR42ZGF4VKMDXJ47BSPU3Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.536847771 (594h58m30.407930806s) [noise=true , address=77.90.31.19:4859 , piece_id=HEZMENV33Y6A6MMMMDIUECW44662O24WTIG2RWCWZL3IGNKWCH2A, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.680580563 (594h58m30.551663608s) [noise=true , address=45.91.116.12:4253 , piece_id=NPLIO7NLEIRTQAWDQ7WZVZE5ZSRLDPDNM34M6SEI53G5XEHB3LJA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.065501378 (594h58m30.936584414s) [noise=true , address=145.79.227.49:4359 , piece_id=RGCUJ32JS6NXVVLH7HWUD5IZNV3HZPRRVIHEBJEII2JPEENVIKZA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.104555514 (594h58m30.975638559s) [noise=true , address=145.79.227.44:4219 , piece_id=NSHMZDOI264JBC5ML4GRTH66FKT76C5QQ534UA4QDFEIG5P6CHDQ, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.115841635 (594h58m30.98692467s) [noise=true , address=145.79.227.30:4783 , piece_id=UIXZYYXWC7KJZUV3BNZAIUSZNDEYA5RFFLKMXWF3XLNKJF5K2A6Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.364792710 (594h58m31.235875755s) [noise=true , address=145.79.227.40:4062 , piece_id=RTSWJJIEADDCF5Z4JMT4NERSF7EXCWP7QZKDN4KIWEX54UDLNZEA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.374552906 (594h58m31.245635941s) [noise=true , address=145.79.227.43:4179 , piece_id=CQ32O4X3PKQN37OWC3WU5Z5FF7CCAIOAWLUISOXJTTIXTSMKYNQA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.374572984 (594h58m31.245656019s) [seg_pos={0 0}, upload=521557]: Segment reached optimal threshold of 14 pieces. Dec 30 10:27:31.374775900 (594h58m31.245858935s) [noise=true , address=45.91.116.10:4115 , piece_id=DRZ2YOFTPSE7SQD6MJ6YQWRQN6MDOLHHP7LWJMKMCPHO37A72C4A, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1arcuTW5rwnvfiqi2H7dL3oL6jKBkDeGq6DmgRGpS9Vs9BCHmf): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.374782553 (594h58m31.245865598s) [noise=true , address=45.91.116.10:4115 , piece_id=DRZ2YOFTPSE7SQD6MJ6YQWRQN6MDOLHHP7LWJMKMCPHO37A72C4A, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.374972405 (594h58m31.24605544s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.374978286 (594h58m31.246061321s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375019294 (594h58m31.246102349s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.375038921 (594h58m31.246121966s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375091411 (594h58m31.246174446s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE): protocol: write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection; write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection Dec 30 10:27:31.375099606 (594h58m31.246182641s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375195900 (594h58m31.246278935s) [seg_pos={0 0}, upload=521557]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:27:31.375261705 (594h58m31.24634474s) [upload=521557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:27:31.375266604 (594h58m31.246349639s) [upload=521557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:27:31.664767884 (594h58m31.53585099s) [upload=521557]: Done uploading segments... Dec 30 10:27:31.664793884 (594h58m31.535876929s) [upload=521557]: Upload finished. err: <nil> Duration: 36.498669745s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [...X......X...X...X........X.....X.X............X......X.X.X] duration: 5.001722384s trace: https://jaeger.dev.storj.io/trace/58c2b839620d74e0 FAIL (2025-12-30 10:38:29 UTC) orbiter warning: context deadline exceeded Duration: 5.00180866s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [......XX.X.XX.X.X.X.XX.X.....XX..........XX...X.X..XX.XX....] duration: 8.104190895s trace: https://jaeger.dev.storj.io/trace/495530043a5329b3 FAIL (2025-12-30 10:38:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: M7X6D4CJSGVAKXBETA325RFNF7766XPWVVIBUWDTCKPXUW2KU6DQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: 3PAEOVMQUSNSDMHYIHZEIHKRUGCQF3XI3CXYWNYH4MQJX7W3UG3A) context deadline exceeded ======LOGS====== Dec 30 10:35:37.649698347 (595h6m37.520781382s) [upload=521677]: Starting upload Dec 30 10:35:37.649784342 (595h6m37.520867397s) [upload=521677]: Done starting upload Dec 30 10:35:37.649861779 (595h6m37.520944814s) [upload=521677]: Uploading segments... Dec 30 10:35:37.650077299 (595h6m37.521160334s) [upload=521677]: Got next segment. Inline: false Dec 30 10:35:37.650127594 (595h6m37.521210639s) [upload=521677]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:35:37.650133907 (595h6m37.521216952s) [upload=521677]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:35:37.898044742 (595h6m37.769127777s) [seg_pos={0 0}, upload=521677]: Begin upload segment... Dec 30 10:35:37.898095969 (595h6m37.769179004s) [seg_pos={0 0}, upload=521677]: Done begin upload segment. Dec 30 10:35:37.898126226 (595h6m37.769209261s) [noise=true , address=145.79.227.28:4726 , piece_id=USLT7VCJXW3XIUBKJIWEZXDQW3UCNU7OBJE5W5FA656CV7Z7RBJQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.898191080 (595h6m37.769274115s) [noise=true , address=77.90.31.13:4626 , piece_id=M7X6D4CJSGVAKXBETA325RFNF7766XPWVVIBUWDTCKPXUW2KU6DQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.898892445 (595h6m37.76997548s) [noise=true , address=145.79.227.51:4409 , piece_id=CTBBMYJHQR6QKV234FMEWOXHOSDZFJCOBGIRBHTHQHCG63UO754A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.898908826 (595h6m37.769991861s) [noise=true , address=145.79.227.36:4519 , piece_id=274CAK6P7CEUKT6AZGS2IXVTPZVLVFDJ5ZSW6I5J5PCXZLVLCLAQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899241260 (595h6m37.770324305s) [noise=true , address=145.79.227.15:4873 , piece_id=V2LX57CMLFCHSI74REOYCTBYV2XASSNJ3MFSURWZHNDPH2GBLKYA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899249946 (595h6m37.770332981s) [noise=true , address=145.79.227.19:4079 , piece_id=NHL2UHODXNQOKXPVQMT3DDFGLE4ON3EJZCJW2HSXCVQKYM36XWDA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899499791 (595h6m37.770582826s) [noise=true , address=145.79.227.5:4221 , piece_id=A5U5FCSJB3JXSRALH2GJITQFWMKRP23PHDXI4D6FZMJPYTWGTALQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899570246 (595h6m37.770653281s) [noise=true , address=45.91.116.13:4153 , piece_id=3PAEOVMQUSNSDMHYIHZEIHKRUGCQF3XI3CXYWNYH4MQJX7W3UG3A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899758805 (595h6m37.77084184s) [noise=true , address=145.79.227.12:4765 , piece_id=HXLI3IRM2WLLXQZZZJDE7XEVDSMEXIAXJHPSDF3XI6AVE75E3SVA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899913429 (595h6m37.770996464s) [noise=true , address=145.79.227.22:4510 , piece_id=NPW6KWE3SIYWVUN4WSU5BO2BTEF3S5CYSTPDBBLG52Y6B4RF72QQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900031054 (595h6m37.771114089s) [noise=true , address=145.79.227.18:4006 , piece_id=MLZZKCQFU4ZBX4J5FUZHJ26CWSOMSJJHOSDQLGB37GWVHYHXNDUA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900277222 (595h6m37.771360258s) [noise=true , address=45.91.116.4:4937 , piece_id=IALTP3UKI4PULI2VVKPPTYY5Z2CGWZW3C5V3TYJC3I7G7LUZLM5A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900348268 (595h6m37.771431313s) [noise=true , address=145.79.227.34:4905 , piece_id=DFIORHRM57H5XRZGVIQYL3V4E4AGCWM7C7X2P7QZTWNW6GVQZVLQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900602161 (595h6m37.771685206s) [noise=true , address=145.79.227.7:4295 , piece_id=4RQS6473BFZSNAJD2VVEJNY2D52CVTLZKQ4MBEEJM53AUOL5TRSQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900657366 (595h6m37.771740401s) [noise=true , address=145.79.227.40:4050 , piece_id=YNGUQV7LIGRE2UZPSZXWHTAMZJE4GY2Y6WRKNHJ2LZWDWKEKLARQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.901035526 (595h6m37.772118571s) [noise=true , address=45.91.116.7:4012 , piece_id=T72JNTV4TUB23CC5XRPYWJZ5UCRE77WJHGAYM4SK2BAFWTDADY2A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:39.604350149 (595h6m39.475433194s) [upload=521677]: Next returned nil segment Dec 30 10:35:39.604382581 (595h6m39.475465616s) [upload=521677]: Waiting for error group managing segments... Dec 30 10:35:39.877515232 (595h6m39.748598267s) [noise=true , address=145.79.227.12:4765 , piece_id=HXLI3IRM2WLLXQZZZJDE7XEVDSMEXIAXJHPSDF3XI6AVE75E3SVA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.882503961 (595h6m39.753586996s) [noise=true , address=145.79.227.15:4873 , piece_id=V2LX57CMLFCHSI74REOYCTBYV2XASSNJ3MFSURWZHNDPH2GBLKYA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.888169319 (595h6m39.759252355s) [noise=true , address=145.79.227.19:4079 , piece_id=NHL2UHODXNQOKXPVQMT3DDFGLE4ON3EJZCJW2HSXCVQKYM36XWDA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.888523183 (595h6m39.759606228s) [noise=true , address=145.79.227.22:4510 , piece_id=NPW6KWE3SIYWVUN4WSU5BO2BTEF3S5CYSTPDBBLG52Y6B4RF72QQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.901555459 (595h6m39.772638505s) [noise=true , address=145.79.227.36:4519 , piece_id=274CAK6P7CEUKT6AZGS2IXVTPZVLVFDJ5ZSW6I5J5PCXZLVLCLAQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.904746936 (595h6m39.775829981s) [noise=true , address=145.79.227.51:4409 , piece_id=CTBBMYJHQR6QKV234FMEWOXHOSDZFJCOBGIRBHTHQHCG63UO754A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.910736492 (595h6m39.781819537s) [noise=true , address=145.79.227.7:4295 , piece_id=4RQS6473BFZSNAJD2VVEJNY2D52CVTLZKQ4MBEEJM53AUOL5TRSQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.974959302 (595h6m39.846042337s) [noise=true , address=145.79.227.5:4221 , piece_id=A5U5FCSJB3JXSRALH2GJITQFWMKRP23PHDXI4D6FZMJPYTWGTALQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.981452888 (595h6m39.852535923s) [noise=true , address=145.79.227.18:4006 , piece_id=MLZZKCQFU4ZBX4J5FUZHJ26CWSOMSJJHOSDQLGB37GWVHYHXNDUA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.982670747 (595h6m39.853753782s) [noise=true , address=145.79.227.28:4726 , piece_id=USLT7VCJXW3XIUBKJIWEZXDQW3UCNU7OBJE5W5FA656CV7Z7RBJQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.209859583 (595h6m40.080942618s) [noise=true , address=45.91.116.13:4153 , piece_id=3PAEOVMQUSNSDMHYIHZEIHKRUGCQF3XI3CXYWNYH4MQJX7W3UG3A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.287338457 (595h6m40.158421492s) [noise=true , address=45.91.116.4:4937 , piece_id=IALTP3UKI4PULI2VVKPPTYY5Z2CGWZW3C5V3TYJC3I7G7LUZLM5A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.500418026 (595h6m40.371501071s) [noise=true , address=77.90.31.13:4626 , piece_id=M7X6D4CJSGVAKXBETA325RFNF7766XPWVVIBUWDTCKPXUW2KU6DQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.500440168 (595h6m40.371523203s) [seg_pos={0 0}, upload=521677]: Segment reached optimal threshold of 13 pieces. Dec 30 10:35:40.500702026 (595h6m40.371785092s) [noise=true , address=145.79.227.34:4905 , piece_id=DFIORHRM57H5XRZGVIQYL3V4E4AGCWM7C7X2P7QZTWNW6GVQZVLQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 30 10:35:40.500712136 (595h6m40.371795181s) [noise=true , address=145.79.227.34:4905 , piece_id=DFIORHRM57H5XRZGVIQYL3V4E4AGCWM7C7X2P7QZTWNW6GVQZVLQ, seg_pos={0 0}, upload=521677]: Long tail cancellation due to optimal threshold Dec 30 10:35:40.500795284 (595h6m40.371878329s) [noise=true , address=145.79.227.40:4050 , piece_id=YNGUQV7LIGRE2UZPSZXWHTAMZJE4GY2Y6WRKNHJ2LZWDWKEKLARQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 10:35:40.500824300 (595h6m40.371907345s) [noise=true , address=145.79.227.40:4050 , piece_id=YNGUQV7LIGRE2UZPSZXWHTAMZJE4GY2Y6WRKNHJ2LZWDWKEKLARQ, seg_pos={0 0}, upload=521677]: Long tail cancellation due to optimal threshold Dec 30 10:35:40.501039109 (595h6m40.372122144s) [noise=true , address=45.91.116.7:4012 , piece_id=T72JNTV4TUB23CC5XRPYWJZ5UCRE77WJHGAYM4SK2BAFWTDADY2A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 30 10:35:40.501047384 (595h6m40.372130419s) [noise=true , address=45.91.116.7:4012 , piece_id=T72JNTV4TUB23CC5XRPYWJZ5UCRE77WJHGAYM4SK2BAFWTDADY2A, seg_pos={0 0}, upload=521677]: Long tail cancellation due to optimal threshold Dec 30 10:35:40.501068395 (595h6m40.37215144s) [seg_pos={0 0}, upload=521677]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:35:40.501138688 (595h6m40.372221733s) [upload=521677]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:35:40.501143888 (595h6m40.372226933s) [upload=521677]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:35:40.800282569 (595h6m40.671365605s) [upload=521677]: Done uploading segments... Dec 30 10:35:40.800309010 (595h6m40.671392045s) [upload=521677]: Upload finished. err: <nil> Duration: 8.104213928s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.......XX.X.XX.X.X.X.XX.X.....XX..........XX...X.X..XX.XX...] duration: 5.810521215s trace: https://jaeger.dev.storj.io/trace/495530043a5329b3 FAIL (2025-12-30 10:37:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: M7X6D4CJSGVAKXBETA325RFNF7766XPWVVIBUWDTCKPXUW2KU6DQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: 3PAEOVMQUSNSDMHYIHZEIHKRUGCQF3XI3CXYWNYH4MQJX7W3UG3A) context deadline exceeded ======LOGS====== Dec 30 10:35:37.649698347 (595h6m37.520781382s) [upload=521677]: Starting upload Dec 30 10:35:37.649784342 (595h6m37.520867397s) [upload=521677]: Done starting upload Dec 30 10:35:37.649861779 (595h6m37.520944814s) [upload=521677]: Uploading segments... Dec 30 10:35:37.650077299 (595h6m37.521160334s) [upload=521677]: Got next segment. Inline: false Dec 30 10:35:37.650127594 (595h6m37.521210639s) [upload=521677]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:35:37.650133907 (595h6m37.521216952s) [upload=521677]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:35:37.898044742 (595h6m37.769127777s) [seg_pos={0 0}, upload=521677]: Begin upload segment... Dec 30 10:35:37.898095969 (595h6m37.769179004s) [seg_pos={0 0}, upload=521677]: Done begin upload segment. Dec 30 10:35:37.898126226 (595h6m37.769209261s) [noise=true , address=145.79.227.28:4726 , piece_id=USLT7VCJXW3XIUBKJIWEZXDQW3UCNU7OBJE5W5FA656CV7Z7RBJQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.898191080 (595h6m37.769274115s) [noise=true , address=77.90.31.13:4626 , piece_id=M7X6D4CJSGVAKXBETA325RFNF7766XPWVVIBUWDTCKPXUW2KU6DQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.898892445 (595h6m37.76997548s) [noise=true , address=145.79.227.51:4409 , piece_id=CTBBMYJHQR6QKV234FMEWOXHOSDZFJCOBGIRBHTHQHCG63UO754A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.898908826 (595h6m37.769991861s) [noise=true , address=145.79.227.36:4519 , piece_id=274CAK6P7CEUKT6AZGS2IXVTPZVLVFDJ5ZSW6I5J5PCXZLVLCLAQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899241260 (595h6m37.770324305s) [noise=true , address=145.79.227.15:4873 , piece_id=V2LX57CMLFCHSI74REOYCTBYV2XASSNJ3MFSURWZHNDPH2GBLKYA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899249946 (595h6m37.770332981s) [noise=true , address=145.79.227.19:4079 , piece_id=NHL2UHODXNQOKXPVQMT3DDFGLE4ON3EJZCJW2HSXCVQKYM36XWDA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899499791 (595h6m37.770582826s) [noise=true , address=145.79.227.5:4221 , piece_id=A5U5FCSJB3JXSRALH2GJITQFWMKRP23PHDXI4D6FZMJPYTWGTALQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899570246 (595h6m37.770653281s) [noise=true , address=45.91.116.13:4153 , piece_id=3PAEOVMQUSNSDMHYIHZEIHKRUGCQF3XI3CXYWNYH4MQJX7W3UG3A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899758805 (595h6m37.77084184s) [noise=true , address=145.79.227.12:4765 , piece_id=HXLI3IRM2WLLXQZZZJDE7XEVDSMEXIAXJHPSDF3XI6AVE75E3SVA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899913429 (595h6m37.770996464s) [noise=true , address=145.79.227.22:4510 , piece_id=NPW6KWE3SIYWVUN4WSU5BO2BTEF3S5CYSTPDBBLG52Y6B4RF72QQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900031054 (595h6m37.771114089s) [noise=true , address=145.79.227.18:4006 , piece_id=MLZZKCQFU4ZBX4J5FUZHJ26CWSOMSJJHOSDQLGB37GWVHYHXNDUA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900277222 (595h6m37.771360258s) [noise=true , address=45.91.116.4:4937 , piece_id=IALTP3UKI4PULI2VVKPPTYY5Z2CGWZW3C5V3TYJC3I7G7LUZLM5A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900348268 (595h6m37.771431313s) [noise=true , address=145.79.227.34:4905 , piece_id=DFIORHRM57H5XRZGVIQYL3V4E4AGCWM7C7X2P7QZTWNW6GVQZVLQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900602161 (595h6m37.771685206s) [noise=true , address=145.79.227.7:4295 , piece_id=4RQS6473BFZSNAJD2VVEJNY2D52CVTLZKQ4MBEEJM53AUOL5TRSQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900657366 (595h6m37.771740401s) [noise=true , address=145.79.227.40:4050 , piece_id=YNGUQV7LIGRE2UZPSZXWHTAMZJE4GY2Y6WRKNHJ2LZWDWKEKLARQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.901035526 (595h6m37.772118571s) [noise=true , address=45.91.116.7:4012 , piece_id=T72JNTV4TUB23CC5XRPYWJZ5UCRE77WJHGAYM4SK2BAFWTDADY2A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:39.604350149 (595h6m39.475433194s) [upload=521677]: Next returned nil segment Dec 30 10:35:39.604382581 (595h6m39.475465616s) [upload=521677]: Waiting for error group managing segments... Dec 30 10:35:39.877515232 (595h6m39.748598267s) [noise=true , address=145.79.227.12:4765 , piece_id=HXLI3IRM2WLLXQZZZJDE7XEVDSMEXIAXJHPSDF3XI6AVE75E3SVA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.882503961 (595h6m39.753586996s) [noise=true , address=145.79.227.15:4873 , piece_id=V2LX57CMLFCHSI74REOYCTBYV2XASSNJ3MFSURWZHNDPH2GBLKYA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.888169319 (595h6m39.759252355s) [noise=true , address=145.79.227.19:4079 , piece_id=NHL2UHODXNQOKXPVQMT3DDFGLE4ON3EJZCJW2HSXCVQKYM36XWDA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.888523183 (595h6m39.759606228s) [noise=true , address=145.79.227.22:4510 , piece_id=NPW6KWE3SIYWVUN4WSU5BO2BTEF3S5CYSTPDBBLG52Y6B4RF72QQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.901555459 (595h6m39.772638505s) [noise=true , address=145.79.227.36:4519 , piece_id=274CAK6P7CEUKT6AZGS2IXVTPZVLVFDJ5ZSW6I5J5PCXZLVLCLAQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.904746936 (595h6m39.775829981s) [noise=true , address=145.79.227.51:4409 , piece_id=CTBBMYJHQR6QKV234FMEWOXHOSDZFJCOBGIRBHTHQHCG63UO754A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.910736492 (595h6m39.781819537s) [noise=true , address=145.79.227.7:4295 , piece_id=4RQS6473BFZSNAJD2VVEJNY2D52CVTLZKQ4MBEEJM53AUOL5TRSQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.974959302 (595h6m39.846042337s) [noise=true , address=145.79.227.5:4221 , piece_id=A5U5FCSJB3JXSRALH2GJITQFWMKRP23PHDXI4D6FZMJPYTWGTALQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.981452888 (595h6m39.852535923s) [noise=true , address=145.79.227.18:4006 , piece_id=MLZZKCQFU4ZBX4J5FUZHJ26CWSOMSJJHOSDQLGB37GWVHYHXNDUA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.982670747 (595h6m39.853753782s) [noise=true , address=145.79.227.28:4726 , piece_id=USLT7VCJXW3XIUBKJIWEZXDQW3UCNU7OBJE5W5FA656CV7Z7RBJQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.209859583 (595h6m40.080942618s) [noise=true , address=45.91.116.13:4153 , piece_id=3PAEOVMQUSNSDMHYIHZEIHKRUGCQF3XI3CXYWNYH4MQJX7W3UG3A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.287338457 (595h6m40.158421492s) [noise=true , address=45.91.116.4:4937 , piece_id=IALTP3UKI4PULI2VVKPPTYY5Z2CGWZW3C5V3TYJC3I7G7LUZLM5A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.500418026 (595h6m40.371501071s) [noise=true , address=77.90.31.13:4626 , piece_id=M7X6D4CJSGVAKXBETA325RFNF7766XPWVVIBUWDTCKPXUW2KU6DQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.500440168 (595h6m40.371523203s) [seg_pos={0 0}, upload=521677]: Segment reached optimal threshold of 13 pieces. Dec 30 10:35:40.500702026 (595h6m40.371785092s) [noise=true , address=145.79.227.34:4905 , piece_id=DFIORHRM57H5XRZGVIQYL3V4E4AGCWM7C7X2P7QZTWNW6GVQZVLQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 30 10:35:40.500712136 (595h6m40.371795181s) [noise=true , address=145.79.227.34:4905 , piece_id=DFIORHRM57H5XRZGVIQYL3V4E4AGCWM7C7X2P7QZTWNW6GVQZVLQ, seg_pos={0 0}, upload=521677]: Long tail cancellation due to optimal threshold Dec 30 10:35:40.500795284 (595h6m40.371878329s) [noise=true , address=145.79.227.40:4050 , piece_id=YNGUQV7LIGRE2UZPSZXWHTAMZJE4GY2Y6WRKNHJ2LZWDWKEKLARQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 10:35:40.500824300 (595h6m40.371907345s) [noise=true , address=145.79.227.40:4050 , piece_id=YNGUQV7LIGRE2UZPSZXWHTAMZJE4GY2Y6WRKNHJ2LZWDWKEKLARQ, seg_pos={0 0}, upload=521677]: Long tail cancellation due to optimal threshold Dec 30 10:35:40.501039109 (595h6m40.372122144s) [noise=true , address=45.91.116.7:4012 , piece_id=T72JNTV4TUB23CC5XRPYWJZ5UCRE77WJHGAYM4SK2BAFWTDADY2A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 30 10:35:40.501047384 (595h6m40.372130419s) [noise=true , address=45.91.116.7:4012 , piece_id=T72JNTV4TUB23CC5XRPYWJZ5UCRE77WJHGAYM4SK2BAFWTDADY2A, seg_pos={0 0}, upload=521677]: Long tail cancellation due to optimal threshold Dec 30 10:35:40.501068395 (595h6m40.37215144s) [seg_pos={0 0}, upload=521677]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:35:40.501138688 (595h6m40.372221733s) [upload=521677]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:35:40.501143888 (595h6m40.372226933s) [upload=521677]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:35:40.800282569 (595h6m40.671365605s) [upload=521677]: Done uploading segments... Dec 30 10:35:40.800309010 (595h6m40.671392045s) [upload=521677]: Upload finished. err: <nil> Duration: 5.810546073s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [....X......X...X...X........X.....X.X............X......X.X.] duration: 3.774636032s trace: https://jaeger.dev.storj.io/trace/7f92df70254831e9 FAIL (2025-12-30 10:37:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.774665619s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [X...............................X....X.........X............] duration: 27.126140236s trace: https://jaeger.dev.storj.io/trace/1dc8263b2e55d7dc FAIL (2025-12-30 10:37:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311300688 (594h58m28.182383733s) [noise=true , address=145.79.227.43:4179 , piece_id=CQ32O4X3PKQN37OWC3WU5Z5FF7CCAIOAWLUISOXJTTIXTSMKYNQA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311642950 (594h58m28.182725995s) [noise=true , address=145.79.227.30:4783 , piece_id=UIXZYYXWC7KJZUV3BNZAIUSZNDEYA5RFFLKMXWF3XLNKJF5K2A6Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311712022 (594h58m28.182795057s) [noise=true , address=145.79.227.35:4492 , piece_id=GM7TCP65VSFAGWDCRG4VKWAPJHU5HGZFZTVAD4XM6PJFMLCC4NRA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312038483 (594h58m28.183121528s) [noise=true , address=77.90.31.17:4798 , piece_id=HT7E2VVRR2IURW5OIXQ4BLR55QI2BEKJG3VIRG4CABVPCUZRIGWA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312100291 (594h58m28.183183326s) [noise=true , address=77.90.31.19:4859 , piece_id=HEZMENV33Y6A6MMMMDIUECW44662O24WTIG2RWCWZL3IGNKWCH2A, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312494201 (594h58m28.183577246s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312500252 (594h58m28.183583287s) [noise=true , address=145.79.227.49:4359 , piece_id=RGCUJ32JS6NXVVLH7HWUD5IZNV3HZPRRVIHEBJEII2JPEENVIKZA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312899192 (594h58m28.183982227s) [noise=true , address=145.79.227.18:4037 , piece_id=IQZILX3EVK57WA5QGC4VGRYTYWQX37PLIASYIYXIHWBWKGTSEESA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313265008 (594h58m28.184348053s) [noise=true , address=145.79.227.6:4267 , piece_id=ZE5IVJYY2XEW3QOZBBE7LYICAALYG3DZHU6K7UDYUGBHRTWPQYPA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313640203 (594h58m28.184723248s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313904957 (594h58m28.184988012s) [noise=true , address=45.91.116.14:4213 , piece_id=BUTDQ3PI56Q7ZHNYHCEMWPHJUI42LXBR42ZGF4VKMDXJ47BSPU3Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314038000 (594h58m28.185121045s) [noise=true , address=77.90.31.12:4603 , piece_id=QQ5V6RXCHOHVL2QTWLLRFPJBAQQ5KW47T2N44EBMIUW747Q4ZOAA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314410920 (594h58m28.185493965s) [noise=true , address=145.79.227.44:4219 , piece_id=NSHMZDOI264JBC5ML4GRTH66FKT76C5QQ534UA4QDFEIG5P6CHDQ, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314664292 (594h58m28.185747338s) [noise=true , address=145.79.227.40:4062 , piece_id=RTSWJJIEADDCF5Z4JMT4NERSF7EXCWP7QZKDN4KIWEX54UDLNZEA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.315164305 (594h58m28.18624734s) [noise=true , address=77.90.31.10:4540 , piece_id=5LXEPFE5MNCGS6KJK4RP5Q5MV4M3TQMVR5FK4DYTYY4MKTFXTP2Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:29.778852769 (594h58m29.649935804s) [upload=521557]: Next returned nil segment Dec 30 10:27:29.778904818 (594h58m29.649987863s) [upload=521557]: Waiting for error group managing segments... Dec 30 10:27:30.177976438 (594h58m30.049059474s) [noise=true , address=145.79.227.18:4037 , piece_id=IQZILX3EVK57WA5QGC4VGRYTYWQX37PLIASYIYXIHWBWKGTSEESA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.239401413 (594h58m30.110484448s) [noise=true , address=145.79.227.6:4267 , piece_id=ZE5IVJYY2XEW3QOZBBE7LYICAALYG3DZHU6K7UDYUGBHRTWPQYPA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.247231673 (594h58m30.118314708s) [noise=true , address=145.79.227.35:4492 , piece_id=GM7TCP65VSFAGWDCRG4VKWAPJHU5HGZFZTVAD4XM6PJFMLCC4NRA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.364460186 (594h58m30.235543221s) [noise=true , address=77.90.31.17:4798 , piece_id=HT7E2VVRR2IURW5OIXQ4BLR55QI2BEKJG3VIRG4CABVPCUZRIGWA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.415614833 (594h58m30.286697868s) [noise=true , address=77.90.31.12:4603 , piece_id=QQ5V6RXCHOHVL2QTWLLRFPJBAQQ5KW47T2N44EBMIUW747Q4ZOAA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.444886882 (594h58m30.315969917s) [noise=true , address=77.90.31.10:4540 , piece_id=5LXEPFE5MNCGS6KJK4RP5Q5MV4M3TQMVR5FK4DYTYY4MKTFXTP2Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.512047427 (594h58m30.383130462s) [noise=true , address=45.91.116.14:4213 , piece_id=BUTDQ3PI56Q7ZHNYHCEMWPHJUI42LXBR42ZGF4VKMDXJ47BSPU3Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.536847771 (594h58m30.407930806s) [noise=true , address=77.90.31.19:4859 , piece_id=HEZMENV33Y6A6MMMMDIUECW44662O24WTIG2RWCWZL3IGNKWCH2A, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.680580563 (594h58m30.551663608s) [noise=true , address=45.91.116.12:4253 , piece_id=NPLIO7NLEIRTQAWDQ7WZVZE5ZSRLDPDNM34M6SEI53G5XEHB3LJA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.065501378 (594h58m30.936584414s) [noise=true , address=145.79.227.49:4359 , piece_id=RGCUJ32JS6NXVVLH7HWUD5IZNV3HZPRRVIHEBJEII2JPEENVIKZA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.104555514 (594h58m30.975638559s) [noise=true , address=145.79.227.44:4219 , piece_id=NSHMZDOI264JBC5ML4GRTH66FKT76C5QQ534UA4QDFEIG5P6CHDQ, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.115841635 (594h58m30.98692467s) [noise=true , address=145.79.227.30:4783 , piece_id=UIXZYYXWC7KJZUV3BNZAIUSZNDEYA5RFFLKMXWF3XLNKJF5K2A6Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.364792710 (594h58m31.235875755s) [noise=true , address=145.79.227.40:4062 , piece_id=RTSWJJIEADDCF5Z4JMT4NERSF7EXCWP7QZKDN4KIWEX54UDLNZEA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.374552906 (594h58m31.245635941s) [noise=true , address=145.79.227.43:4179 , piece_id=CQ32O4X3PKQN37OWC3WU5Z5FF7CCAIOAWLUISOXJTTIXTSMKYNQA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.374572984 (594h58m31.245656019s) [seg_pos={0 0}, upload=521557]: Segment reached optimal threshold of 14 pieces. Dec 30 10:27:31.374775900 (594h58m31.245858935s) [noise=true , address=45.91.116.10:4115 , piece_id=DRZ2YOFTPSE7SQD6MJ6YQWRQN6MDOLHHP7LWJMKMCPHO37A72C4A, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1arcuTW5rwnvfiqi2H7dL3oL6jKBkDeGq6DmgRGpS9Vs9BCHmf): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.374782553 (594h58m31.245865598s) [noise=true , address=45.91.116.10:4115 , piece_id=DRZ2YOFTPSE7SQD6MJ6YQWRQN6MDOLHHP7LWJMKMCPHO37A72C4A, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.374972405 (594h58m31.24605544s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.374978286 (594h58m31.246061321s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375019294 (594h58m31.246102349s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.375038921 (594h58m31.246121966s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375091411 (594h58m31.246174446s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE): protocol: write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection; write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection Dec 30 10:27:31.375099606 (594h58m31.246182641s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375195900 (594h58m31.246278935s) [seg_pos={0 0}, upload=521557]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:27:31.375261705 (594h58m31.24634474s) [upload=521557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:27:31.375266604 (594h58m31.246349639s) [upload=521557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:27:31.664767884 (594h58m31.53585099s) [upload=521557]: Done uploading segments... Dec 30 10:27:31.664793884 (594h58m31.535876929s) [upload=521557]: Upload finished. err: <nil> Duration: 27.12615786s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [........XX.X.XX.X.X.X.XX.X.....XX..........XX...X.X..XX.XX..] duration: 5.972252288s trace: https://jaeger.dev.storj.io/trace/495530043a5329b3 FAIL (2025-12-30 10:36:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: M7X6D4CJSGVAKXBETA325RFNF7766XPWVVIBUWDTCKPXUW2KU6DQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: 3PAEOVMQUSNSDMHYIHZEIHKRUGCQF3XI3CXYWNYH4MQJX7W3UG3A) context deadline exceeded ======LOGS====== Dec 30 10:35:37.649698347 (595h6m37.520781382s) [upload=521677]: Starting upload Dec 30 10:35:37.649784342 (595h6m37.520867397s) [upload=521677]: Done starting upload Dec 30 10:35:37.649861779 (595h6m37.520944814s) [upload=521677]: Uploading segments... Dec 30 10:35:37.650077299 (595h6m37.521160334s) [upload=521677]: Got next segment. Inline: false Dec 30 10:35:37.650127594 (595h6m37.521210639s) [upload=521677]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:35:37.650133907 (595h6m37.521216952s) [upload=521677]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:35:37.898044742 (595h6m37.769127777s) [seg_pos={0 0}, upload=521677]: Begin upload segment... Dec 30 10:35:37.898095969 (595h6m37.769179004s) [seg_pos={0 0}, upload=521677]: Done begin upload segment. Dec 30 10:35:37.898126226 (595h6m37.769209261s) [noise=true , address=145.79.227.28:4726 , piece_id=USLT7VCJXW3XIUBKJIWEZXDQW3UCNU7OBJE5W5FA656CV7Z7RBJQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.898191080 (595h6m37.769274115s) [noise=true , address=77.90.31.13:4626 , piece_id=M7X6D4CJSGVAKXBETA325RFNF7766XPWVVIBUWDTCKPXUW2KU6DQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.898892445 (595h6m37.76997548s) [noise=true , address=145.79.227.51:4409 , piece_id=CTBBMYJHQR6QKV234FMEWOXHOSDZFJCOBGIRBHTHQHCG63UO754A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.898908826 (595h6m37.769991861s) [noise=true , address=145.79.227.36:4519 , piece_id=274CAK6P7CEUKT6AZGS2IXVTPZVLVFDJ5ZSW6I5J5PCXZLVLCLAQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899241260 (595h6m37.770324305s) [noise=true , address=145.79.227.15:4873 , piece_id=V2LX57CMLFCHSI74REOYCTBYV2XASSNJ3MFSURWZHNDPH2GBLKYA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899249946 (595h6m37.770332981s) [noise=true , address=145.79.227.19:4079 , piece_id=NHL2UHODXNQOKXPVQMT3DDFGLE4ON3EJZCJW2HSXCVQKYM36XWDA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899499791 (595h6m37.770582826s) [noise=true , address=145.79.227.5:4221 , piece_id=A5U5FCSJB3JXSRALH2GJITQFWMKRP23PHDXI4D6FZMJPYTWGTALQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899570246 (595h6m37.770653281s) [noise=true , address=45.91.116.13:4153 , piece_id=3PAEOVMQUSNSDMHYIHZEIHKRUGCQF3XI3CXYWNYH4MQJX7W3UG3A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899758805 (595h6m37.77084184s) [noise=true , address=145.79.227.12:4765 , piece_id=HXLI3IRM2WLLXQZZZJDE7XEVDSMEXIAXJHPSDF3XI6AVE75E3SVA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899913429 (595h6m37.770996464s) [noise=true , address=145.79.227.22:4510 , piece_id=NPW6KWE3SIYWVUN4WSU5BO2BTEF3S5CYSTPDBBLG52Y6B4RF72QQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900031054 (595h6m37.771114089s) [noise=true , address=145.79.227.18:4006 , piece_id=MLZZKCQFU4ZBX4J5FUZHJ26CWSOMSJJHOSDQLGB37GWVHYHXNDUA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900277222 (595h6m37.771360258s) [noise=true , address=45.91.116.4:4937 , piece_id=IALTP3UKI4PULI2VVKPPTYY5Z2CGWZW3C5V3TYJC3I7G7LUZLM5A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900348268 (595h6m37.771431313s) [noise=true , address=145.79.227.34:4905 , piece_id=DFIORHRM57H5XRZGVIQYL3V4E4AGCWM7C7X2P7QZTWNW6GVQZVLQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900602161 (595h6m37.771685206s) [noise=true , address=145.79.227.7:4295 , piece_id=4RQS6473BFZSNAJD2VVEJNY2D52CVTLZKQ4MBEEJM53AUOL5TRSQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900657366 (595h6m37.771740401s) [noise=true , address=145.79.227.40:4050 , piece_id=YNGUQV7LIGRE2UZPSZXWHTAMZJE4GY2Y6WRKNHJ2LZWDWKEKLARQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.901035526 (595h6m37.772118571s) [noise=true , address=45.91.116.7:4012 , piece_id=T72JNTV4TUB23CC5XRPYWJZ5UCRE77WJHGAYM4SK2BAFWTDADY2A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:39.604350149 (595h6m39.475433194s) [upload=521677]: Next returned nil segment Dec 30 10:35:39.604382581 (595h6m39.475465616s) [upload=521677]: Waiting for error group managing segments... Dec 30 10:35:39.877515232 (595h6m39.748598267s) [noise=true , address=145.79.227.12:4765 , piece_id=HXLI3IRM2WLLXQZZZJDE7XEVDSMEXIAXJHPSDF3XI6AVE75E3SVA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.882503961 (595h6m39.753586996s) [noise=true , address=145.79.227.15:4873 , piece_id=V2LX57CMLFCHSI74REOYCTBYV2XASSNJ3MFSURWZHNDPH2GBLKYA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.888169319 (595h6m39.759252355s) [noise=true , address=145.79.227.19:4079 , piece_id=NHL2UHODXNQOKXPVQMT3DDFGLE4ON3EJZCJW2HSXCVQKYM36XWDA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.888523183 (595h6m39.759606228s) [noise=true , address=145.79.227.22:4510 , piece_id=NPW6KWE3SIYWVUN4WSU5BO2BTEF3S5CYSTPDBBLG52Y6B4RF72QQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.901555459 (595h6m39.772638505s) [noise=true , address=145.79.227.36:4519 , piece_id=274CAK6P7CEUKT6AZGS2IXVTPZVLVFDJ5ZSW6I5J5PCXZLVLCLAQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.904746936 (595h6m39.775829981s) [noise=true , address=145.79.227.51:4409 , piece_id=CTBBMYJHQR6QKV234FMEWOXHOSDZFJCOBGIRBHTHQHCG63UO754A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.910736492 (595h6m39.781819537s) [noise=true , address=145.79.227.7:4295 , piece_id=4RQS6473BFZSNAJD2VVEJNY2D52CVTLZKQ4MBEEJM53AUOL5TRSQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.974959302 (595h6m39.846042337s) [noise=true , address=145.79.227.5:4221 , piece_id=A5U5FCSJB3JXSRALH2GJITQFWMKRP23PHDXI4D6FZMJPYTWGTALQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.981452888 (595h6m39.852535923s) [noise=true , address=145.79.227.18:4006 , piece_id=MLZZKCQFU4ZBX4J5FUZHJ26CWSOMSJJHOSDQLGB37GWVHYHXNDUA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.982670747 (595h6m39.853753782s) [noise=true , address=145.79.227.28:4726 , piece_id=USLT7VCJXW3XIUBKJIWEZXDQW3UCNU7OBJE5W5FA656CV7Z7RBJQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.209859583 (595h6m40.080942618s) [noise=true , address=45.91.116.13:4153 , piece_id=3PAEOVMQUSNSDMHYIHZEIHKRUGCQF3XI3CXYWNYH4MQJX7W3UG3A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.287338457 (595h6m40.158421492s) [noise=true , address=45.91.116.4:4937 , piece_id=IALTP3UKI4PULI2VVKPPTYY5Z2CGWZW3C5V3TYJC3I7G7LUZLM5A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.500418026 (595h6m40.371501071s) [noise=true , address=77.90.31.13:4626 , piece_id=M7X6D4CJSGVAKXBETA325RFNF7766XPWVVIBUWDTCKPXUW2KU6DQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.500440168 (595h6m40.371523203s) [seg_pos={0 0}, upload=521677]: Segment reached optimal threshold of 13 pieces. Dec 30 10:35:40.500702026 (595h6m40.371785092s) [noise=true , address=145.79.227.34:4905 , piece_id=DFIORHRM57H5XRZGVIQYL3V4E4AGCWM7C7X2P7QZTWNW6GVQZVLQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 30 10:35:40.500712136 (595h6m40.371795181s) [noise=true , address=145.79.227.34:4905 , piece_id=DFIORHRM57H5XRZGVIQYL3V4E4AGCWM7C7X2P7QZTWNW6GVQZVLQ, seg_pos={0 0}, upload=521677]: Long tail cancellation due to optimal threshold Dec 30 10:35:40.500795284 (595h6m40.371878329s) [noise=true , address=145.79.227.40:4050 , piece_id=YNGUQV7LIGRE2UZPSZXWHTAMZJE4GY2Y6WRKNHJ2LZWDWKEKLARQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 10:35:40.500824300 (595h6m40.371907345s) [noise=true , address=145.79.227.40:4050 , piece_id=YNGUQV7LIGRE2UZPSZXWHTAMZJE4GY2Y6WRKNHJ2LZWDWKEKLARQ, seg_pos={0 0}, upload=521677]: Long tail cancellation due to optimal threshold Dec 30 10:35:40.501039109 (595h6m40.372122144s) [noise=true , address=45.91.116.7:4012 , piece_id=T72JNTV4TUB23CC5XRPYWJZ5UCRE77WJHGAYM4SK2BAFWTDADY2A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 30 10:35:40.501047384 (595h6m40.372130419s) [noise=true , address=45.91.116.7:4012 , piece_id=T72JNTV4TUB23CC5XRPYWJZ5UCRE77WJHGAYM4SK2BAFWTDADY2A, seg_pos={0 0}, upload=521677]: Long tail cancellation due to optimal threshold Dec 30 10:35:40.501068395 (595h6m40.37215144s) [seg_pos={0 0}, upload=521677]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:35:40.501138688 (595h6m40.372221733s) [upload=521677]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:35:40.501143888 (595h6m40.372226933s) [upload=521677]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:35:40.800282569 (595h6m40.671365605s) [upload=521677]: Done uploading segments... Dec 30 10:35:40.800309010 (595h6m40.671392045s) [upload=521677]: Upload finished. err: <nil> Duration: 5.972281574s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.....X......X...X...X........X.....X.X............X......X.X] duration: 5.001068397s trace: https://jaeger.dev.storj.io/trace/7f92df70254831e9 FAIL (2025-12-30 10:36:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001093305s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X...............................X....X.........X...........] duration: 43.577780652s trace: https://jaeger.dev.storj.io/trace/1dc8263b2e55d7dc FAIL (2025-12-30 10:36:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311300688 (594h58m28.182383733s) [noise=true , address=145.79.227.43:4179 , piece_id=CQ32O4X3PKQN37OWC3WU5Z5FF7CCAIOAWLUISOXJTTIXTSMKYNQA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311642950 (594h58m28.182725995s) [noise=true , address=145.79.227.30:4783 , piece_id=UIXZYYXWC7KJZUV3BNZAIUSZNDEYA5RFFLKMXWF3XLNKJF5K2A6Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311712022 (594h58m28.182795057s) [noise=true , address=145.79.227.35:4492 , piece_id=GM7TCP65VSFAGWDCRG4VKWAPJHU5HGZFZTVAD4XM6PJFMLCC4NRA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312038483 (594h58m28.183121528s) [noise=true , address=77.90.31.17:4798 , piece_id=HT7E2VVRR2IURW5OIXQ4BLR55QI2BEKJG3VIRG4CABVPCUZRIGWA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312100291 (594h58m28.183183326s) [noise=true , address=77.90.31.19:4859 , piece_id=HEZMENV33Y6A6MMMMDIUECW44662O24WTIG2RWCWZL3IGNKWCH2A, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312494201 (594h58m28.183577246s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312500252 (594h58m28.183583287s) [noise=true , address=145.79.227.49:4359 , piece_id=RGCUJ32JS6NXVVLH7HWUD5IZNV3HZPRRVIHEBJEII2JPEENVIKZA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312899192 (594h58m28.183982227s) [noise=true , address=145.79.227.18:4037 , piece_id=IQZILX3EVK57WA5QGC4VGRYTYWQX37PLIASYIYXIHWBWKGTSEESA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313265008 (594h58m28.184348053s) [noise=true , address=145.79.227.6:4267 , piece_id=ZE5IVJYY2XEW3QOZBBE7LYICAALYG3DZHU6K7UDYUGBHRTWPQYPA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313640203 (594h58m28.184723248s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313904957 (594h58m28.184988012s) [noise=true , address=45.91.116.14:4213 , piece_id=BUTDQ3PI56Q7ZHNYHCEMWPHJUI42LXBR42ZGF4VKMDXJ47BSPU3Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314038000 (594h58m28.185121045s) [noise=true , address=77.90.31.12:4603 , piece_id=QQ5V6RXCHOHVL2QTWLLRFPJBAQQ5KW47T2N44EBMIUW747Q4ZOAA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314410920 (594h58m28.185493965s) [noise=true , address=145.79.227.44:4219 , piece_id=NSHMZDOI264JBC5ML4GRTH66FKT76C5QQ534UA4QDFEIG5P6CHDQ, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314664292 (594h58m28.185747338s) [noise=true , address=145.79.227.40:4062 , piece_id=RTSWJJIEADDCF5Z4JMT4NERSF7EXCWP7QZKDN4KIWEX54UDLNZEA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.315164305 (594h58m28.18624734s) [noise=true , address=77.90.31.10:4540 , piece_id=5LXEPFE5MNCGS6KJK4RP5Q5MV4M3TQMVR5FK4DYTYY4MKTFXTP2Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:29.778852769 (594h58m29.649935804s) [upload=521557]: Next returned nil segment Dec 30 10:27:29.778904818 (594h58m29.649987863s) [upload=521557]: Waiting for error group managing segments... Dec 30 10:27:30.177976438 (594h58m30.049059474s) [noise=true , address=145.79.227.18:4037 , piece_id=IQZILX3EVK57WA5QGC4VGRYTYWQX37PLIASYIYXIHWBWKGTSEESA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.239401413 (594h58m30.110484448s) [noise=true , address=145.79.227.6:4267 , piece_id=ZE5IVJYY2XEW3QOZBBE7LYICAALYG3DZHU6K7UDYUGBHRTWPQYPA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.247231673 (594h58m30.118314708s) [noise=true , address=145.79.227.35:4492 , piece_id=GM7TCP65VSFAGWDCRG4VKWAPJHU5HGZFZTVAD4XM6PJFMLCC4NRA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.364460186 (594h58m30.235543221s) [noise=true , address=77.90.31.17:4798 , piece_id=HT7E2VVRR2IURW5OIXQ4BLR55QI2BEKJG3VIRG4CABVPCUZRIGWA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.415614833 (594h58m30.286697868s) [noise=true , address=77.90.31.12:4603 , piece_id=QQ5V6RXCHOHVL2QTWLLRFPJBAQQ5KW47T2N44EBMIUW747Q4ZOAA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.444886882 (594h58m30.315969917s) [noise=true , address=77.90.31.10:4540 , piece_id=5LXEPFE5MNCGS6KJK4RP5Q5MV4M3TQMVR5FK4DYTYY4MKTFXTP2Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.512047427 (594h58m30.383130462s) [noise=true , address=45.91.116.14:4213 , piece_id=BUTDQ3PI56Q7ZHNYHCEMWPHJUI42LXBR42ZGF4VKMDXJ47BSPU3Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.536847771 (594h58m30.407930806s) [noise=true , address=77.90.31.19:4859 , piece_id=HEZMENV33Y6A6MMMMDIUECW44662O24WTIG2RWCWZL3IGNKWCH2A, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.680580563 (594h58m30.551663608s) [noise=true , address=45.91.116.12:4253 , piece_id=NPLIO7NLEIRTQAWDQ7WZVZE5ZSRLDPDNM34M6SEI53G5XEHB3LJA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.065501378 (594h58m30.936584414s) [noise=true , address=145.79.227.49:4359 , piece_id=RGCUJ32JS6NXVVLH7HWUD5IZNV3HZPRRVIHEBJEII2JPEENVIKZA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.104555514 (594h58m30.975638559s) [noise=true , address=145.79.227.44:4219 , piece_id=NSHMZDOI264JBC5ML4GRTH66FKT76C5QQ534UA4QDFEIG5P6CHDQ, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.115841635 (594h58m30.98692467s) [noise=true , address=145.79.227.30:4783 , piece_id=UIXZYYXWC7KJZUV3BNZAIUSZNDEYA5RFFLKMXWF3XLNKJF5K2A6Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.364792710 (594h58m31.235875755s) [noise=true , address=145.79.227.40:4062 , piece_id=RTSWJJIEADDCF5Z4JMT4NERSF7EXCWP7QZKDN4KIWEX54UDLNZEA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.374552906 (594h58m31.245635941s) [noise=true , address=145.79.227.43:4179 , piece_id=CQ32O4X3PKQN37OWC3WU5Z5FF7CCAIOAWLUISOXJTTIXTSMKYNQA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.374572984 (594h58m31.245656019s) [seg_pos={0 0}, upload=521557]: Segment reached optimal threshold of 14 pieces. Dec 30 10:27:31.374775900 (594h58m31.245858935s) [noise=true , address=45.91.116.10:4115 , piece_id=DRZ2YOFTPSE7SQD6MJ6YQWRQN6MDOLHHP7LWJMKMCPHO37A72C4A, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1arcuTW5rwnvfiqi2H7dL3oL6jKBkDeGq6DmgRGpS9Vs9BCHmf): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.374782553 (594h58m31.245865598s) [noise=true , address=45.91.116.10:4115 , piece_id=DRZ2YOFTPSE7SQD6MJ6YQWRQN6MDOLHHP7LWJMKMCPHO37A72C4A, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.374972405 (594h58m31.24605544s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.374978286 (594h58m31.246061321s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375019294 (594h58m31.246102349s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.375038921 (594h58m31.246121966s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375091411 (594h58m31.246174446s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE): protocol: write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection; write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection Dec 30 10:27:31.375099606 (594h58m31.246182641s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375195900 (594h58m31.246278935s) [seg_pos={0 0}, upload=521557]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:27:31.375261705 (594h58m31.24634474s) [upload=521557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:27:31.375266604 (594h58m31.246349639s) [upload=521557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:27:31.664767884 (594h58m31.53585099s) [upload=521557]: Done uploading segments... Dec 30 10:27:31.664793884 (594h58m31.535876929s) [upload=521557]: Upload finished. err: <nil> Duration: 43.577798758s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.........XX.X.XX.X.X.X.XX.X.....XX..........XX...X.X..XX.XX.] duration: 5.89280164s trace: https://jaeger.dev.storj.io/trace/495530043a5329b3 FAIL (2025-12-30 10:36:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: M7X6D4CJSGVAKXBETA325RFNF7766XPWVVIBUWDTCKPXUW2KU6DQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: 3PAEOVMQUSNSDMHYIHZEIHKRUGCQF3XI3CXYWNYH4MQJX7W3UG3A) context deadline exceeded ======LOGS====== Dec 30 10:35:37.649698347 (595h6m37.520781382s) [upload=521677]: Starting upload Dec 30 10:35:37.649784342 (595h6m37.520867397s) [upload=521677]: Done starting upload Dec 30 10:35:37.649861779 (595h6m37.520944814s) [upload=521677]: Uploading segments... Dec 30 10:35:37.650077299 (595h6m37.521160334s) [upload=521677]: Got next segment. Inline: false Dec 30 10:35:37.650127594 (595h6m37.521210639s) [upload=521677]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:35:37.650133907 (595h6m37.521216952s) [upload=521677]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:35:37.898044742 (595h6m37.769127777s) [seg_pos={0 0}, upload=521677]: Begin upload segment... Dec 30 10:35:37.898095969 (595h6m37.769179004s) [seg_pos={0 0}, upload=521677]: Done begin upload segment. Dec 30 10:35:37.898126226 (595h6m37.769209261s) [noise=true , address=145.79.227.28:4726 , piece_id=USLT7VCJXW3XIUBKJIWEZXDQW3UCNU7OBJE5W5FA656CV7Z7RBJQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.898191080 (595h6m37.769274115s) [noise=true , address=77.90.31.13:4626 , piece_id=M7X6D4CJSGVAKXBETA325RFNF7766XPWVVIBUWDTCKPXUW2KU6DQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.898892445 (595h6m37.76997548s) [noise=true , address=145.79.227.51:4409 , piece_id=CTBBMYJHQR6QKV234FMEWOXHOSDZFJCOBGIRBHTHQHCG63UO754A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.898908826 (595h6m37.769991861s) [noise=true , address=145.79.227.36:4519 , piece_id=274CAK6P7CEUKT6AZGS2IXVTPZVLVFDJ5ZSW6I5J5PCXZLVLCLAQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899241260 (595h6m37.770324305s) [noise=true , address=145.79.227.15:4873 , piece_id=V2LX57CMLFCHSI74REOYCTBYV2XASSNJ3MFSURWZHNDPH2GBLKYA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899249946 (595h6m37.770332981s) [noise=true , address=145.79.227.19:4079 , piece_id=NHL2UHODXNQOKXPVQMT3DDFGLE4ON3EJZCJW2HSXCVQKYM36XWDA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899499791 (595h6m37.770582826s) [noise=true , address=145.79.227.5:4221 , piece_id=A5U5FCSJB3JXSRALH2GJITQFWMKRP23PHDXI4D6FZMJPYTWGTALQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899570246 (595h6m37.770653281s) [noise=true , address=45.91.116.13:4153 , piece_id=3PAEOVMQUSNSDMHYIHZEIHKRUGCQF3XI3CXYWNYH4MQJX7W3UG3A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899758805 (595h6m37.77084184s) [noise=true , address=145.79.227.12:4765 , piece_id=HXLI3IRM2WLLXQZZZJDE7XEVDSMEXIAXJHPSDF3XI6AVE75E3SVA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899913429 (595h6m37.770996464s) [noise=true , address=145.79.227.22:4510 , piece_id=NPW6KWE3SIYWVUN4WSU5BO2BTEF3S5CYSTPDBBLG52Y6B4RF72QQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900031054 (595h6m37.771114089s) [noise=true , address=145.79.227.18:4006 , piece_id=MLZZKCQFU4ZBX4J5FUZHJ26CWSOMSJJHOSDQLGB37GWVHYHXNDUA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900277222 (595h6m37.771360258s) [noise=true , address=45.91.116.4:4937 , piece_id=IALTP3UKI4PULI2VVKPPTYY5Z2CGWZW3C5V3TYJC3I7G7LUZLM5A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900348268 (595h6m37.771431313s) [noise=true , address=145.79.227.34:4905 , piece_id=DFIORHRM57H5XRZGVIQYL3V4E4AGCWM7C7X2P7QZTWNW6GVQZVLQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900602161 (595h6m37.771685206s) [noise=true , address=145.79.227.7:4295 , piece_id=4RQS6473BFZSNAJD2VVEJNY2D52CVTLZKQ4MBEEJM53AUOL5TRSQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900657366 (595h6m37.771740401s) [noise=true , address=145.79.227.40:4050 , piece_id=YNGUQV7LIGRE2UZPSZXWHTAMZJE4GY2Y6WRKNHJ2LZWDWKEKLARQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.901035526 (595h6m37.772118571s) [noise=true , address=45.91.116.7:4012 , piece_id=T72JNTV4TUB23CC5XRPYWJZ5UCRE77WJHGAYM4SK2BAFWTDADY2A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:39.604350149 (595h6m39.475433194s) [upload=521677]: Next returned nil segment Dec 30 10:35:39.604382581 (595h6m39.475465616s) [upload=521677]: Waiting for error group managing segments... Dec 30 10:35:39.877515232 (595h6m39.748598267s) [noise=true , address=145.79.227.12:4765 , piece_id=HXLI3IRM2WLLXQZZZJDE7XEVDSMEXIAXJHPSDF3XI6AVE75E3SVA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.882503961 (595h6m39.753586996s) [noise=true , address=145.79.227.15:4873 , piece_id=V2LX57CMLFCHSI74REOYCTBYV2XASSNJ3MFSURWZHNDPH2GBLKYA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.888169319 (595h6m39.759252355s) [noise=true , address=145.79.227.19:4079 , piece_id=NHL2UHODXNQOKXPVQMT3DDFGLE4ON3EJZCJW2HSXCVQKYM36XWDA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.888523183 (595h6m39.759606228s) [noise=true , address=145.79.227.22:4510 , piece_id=NPW6KWE3SIYWVUN4WSU5BO2BTEF3S5CYSTPDBBLG52Y6B4RF72QQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.901555459 (595h6m39.772638505s) [noise=true , address=145.79.227.36:4519 , piece_id=274CAK6P7CEUKT6AZGS2IXVTPZVLVFDJ5ZSW6I5J5PCXZLVLCLAQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.904746936 (595h6m39.775829981s) [noise=true , address=145.79.227.51:4409 , piece_id=CTBBMYJHQR6QKV234FMEWOXHOSDZFJCOBGIRBHTHQHCG63UO754A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.910736492 (595h6m39.781819537s) [noise=true , address=145.79.227.7:4295 , piece_id=4RQS6473BFZSNAJD2VVEJNY2D52CVTLZKQ4MBEEJM53AUOL5TRSQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.974959302 (595h6m39.846042337s) [noise=true , address=145.79.227.5:4221 , piece_id=A5U5FCSJB3JXSRALH2GJITQFWMKRP23PHDXI4D6FZMJPYTWGTALQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.981452888 (595h6m39.852535923s) [noise=true , address=145.79.227.18:4006 , piece_id=MLZZKCQFU4ZBX4J5FUZHJ26CWSOMSJJHOSDQLGB37GWVHYHXNDUA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.982670747 (595h6m39.853753782s) [noise=true , address=145.79.227.28:4726 , piece_id=USLT7VCJXW3XIUBKJIWEZXDQW3UCNU7OBJE5W5FA656CV7Z7RBJQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.209859583 (595h6m40.080942618s) [noise=true , address=45.91.116.13:4153 , piece_id=3PAEOVMQUSNSDMHYIHZEIHKRUGCQF3XI3CXYWNYH4MQJX7W3UG3A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.287338457 (595h6m40.158421492s) [noise=true , address=45.91.116.4:4937 , piece_id=IALTP3UKI4PULI2VVKPPTYY5Z2CGWZW3C5V3TYJC3I7G7LUZLM5A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.500418026 (595h6m40.371501071s) [noise=true , address=77.90.31.13:4626 , piece_id=M7X6D4CJSGVAKXBETA325RFNF7766XPWVVIBUWDTCKPXUW2KU6DQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.500440168 (595h6m40.371523203s) [seg_pos={0 0}, upload=521677]: Segment reached optimal threshold of 13 pieces. Dec 30 10:35:40.500702026 (595h6m40.371785092s) [noise=true , address=145.79.227.34:4905 , piece_id=DFIORHRM57H5XRZGVIQYL3V4E4AGCWM7C7X2P7QZTWNW6GVQZVLQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 30 10:35:40.500712136 (595h6m40.371795181s) [noise=true , address=145.79.227.34:4905 , piece_id=DFIORHRM57H5XRZGVIQYL3V4E4AGCWM7C7X2P7QZTWNW6GVQZVLQ, seg_pos={0 0}, upload=521677]: Long tail cancellation due to optimal threshold Dec 30 10:35:40.500795284 (595h6m40.371878329s) [noise=true , address=145.79.227.40:4050 , piece_id=YNGUQV7LIGRE2UZPSZXWHTAMZJE4GY2Y6WRKNHJ2LZWDWKEKLARQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 10:35:40.500824300 (595h6m40.371907345s) [noise=true , address=145.79.227.40:4050 , piece_id=YNGUQV7LIGRE2UZPSZXWHTAMZJE4GY2Y6WRKNHJ2LZWDWKEKLARQ, seg_pos={0 0}, upload=521677]: Long tail cancellation due to optimal threshold Dec 30 10:35:40.501039109 (595h6m40.372122144s) [noise=true , address=45.91.116.7:4012 , piece_id=T72JNTV4TUB23CC5XRPYWJZ5UCRE77WJHGAYM4SK2BAFWTDADY2A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 30 10:35:40.501047384 (595h6m40.372130419s) [noise=true , address=45.91.116.7:4012 , piece_id=T72JNTV4TUB23CC5XRPYWJZ5UCRE77WJHGAYM4SK2BAFWTDADY2A, seg_pos={0 0}, upload=521677]: Long tail cancellation due to optimal threshold Dec 30 10:35:40.501068395 (595h6m40.37215144s) [seg_pos={0 0}, upload=521677]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:35:40.501138688 (595h6m40.372221733s) [upload=521677]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:35:40.501143888 (595h6m40.372226933s) [upload=521677]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:35:40.800282569 (595h6m40.671365605s) [upload=521677]: Done uploading segments... Dec 30 10:35:40.800309010 (595h6m40.671392045s) [upload=521677]: Upload finished. err: <nil> Duration: 5.892839283s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..........XX.X.XX.X.X.X.XX.X.....XX..........XX...X.X..XX.XX] duration: 30.003860723s trace: https://jaeger.dev.storj.io/trace/495530043a5329b3 FAIL (2025-12-30 10:36:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, Piece ID: M7X6D4CJSGVAKXBETA325RFNF7766XPWVVIBUWDTCKPXUW2KU6DQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: 3PAEOVMQUSNSDMHYIHZEIHKRUGCQF3XI3CXYWNYH4MQJX7W3UG3A) context deadline exceeded ======LOGS====== Dec 30 10:35:37.649698347 (595h6m37.520781382s) [upload=521677]: Starting upload Dec 30 10:35:37.649784342 (595h6m37.520867397s) [upload=521677]: Done starting upload Dec 30 10:35:37.649861779 (595h6m37.520944814s) [upload=521677]: Uploading segments... Dec 30 10:35:37.650077299 (595h6m37.521160334s) [upload=521677]: Got next segment. Inline: false Dec 30 10:35:37.650127594 (595h6m37.521210639s) [upload=521677]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:35:37.650133907 (595h6m37.521216952s) [upload=521677]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:35:37.898044742 (595h6m37.769127777s) [seg_pos={0 0}, upload=521677]: Begin upload segment... Dec 30 10:35:37.898095969 (595h6m37.769179004s) [seg_pos={0 0}, upload=521677]: Done begin upload segment. Dec 30 10:35:37.898126226 (595h6m37.769209261s) [noise=true , address=145.79.227.28:4726 , piece_id=USLT7VCJXW3XIUBKJIWEZXDQW3UCNU7OBJE5W5FA656CV7Z7RBJQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.898191080 (595h6m37.769274115s) [noise=true , address=77.90.31.13:4626 , piece_id=M7X6D4CJSGVAKXBETA325RFNF7766XPWVVIBUWDTCKPXUW2KU6DQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.898892445 (595h6m37.76997548s) [noise=true , address=145.79.227.51:4409 , piece_id=CTBBMYJHQR6QKV234FMEWOXHOSDZFJCOBGIRBHTHQHCG63UO754A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.898908826 (595h6m37.769991861s) [noise=true , address=145.79.227.36:4519 , piece_id=274CAK6P7CEUKT6AZGS2IXVTPZVLVFDJ5ZSW6I5J5PCXZLVLCLAQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899241260 (595h6m37.770324305s) [noise=true , address=145.79.227.15:4873 , piece_id=V2LX57CMLFCHSI74REOYCTBYV2XASSNJ3MFSURWZHNDPH2GBLKYA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899249946 (595h6m37.770332981s) [noise=true , address=145.79.227.19:4079 , piece_id=NHL2UHODXNQOKXPVQMT3DDFGLE4ON3EJZCJW2HSXCVQKYM36XWDA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899499791 (595h6m37.770582826s) [noise=true , address=145.79.227.5:4221 , piece_id=A5U5FCSJB3JXSRALH2GJITQFWMKRP23PHDXI4D6FZMJPYTWGTALQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899570246 (595h6m37.770653281s) [noise=true , address=45.91.116.13:4153 , piece_id=3PAEOVMQUSNSDMHYIHZEIHKRUGCQF3XI3CXYWNYH4MQJX7W3UG3A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899758805 (595h6m37.77084184s) [noise=true , address=145.79.227.12:4765 , piece_id=HXLI3IRM2WLLXQZZZJDE7XEVDSMEXIAXJHPSDF3XI6AVE75E3SVA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.899913429 (595h6m37.770996464s) [noise=true , address=145.79.227.22:4510 , piece_id=NPW6KWE3SIYWVUN4WSU5BO2BTEF3S5CYSTPDBBLG52Y6B4RF72QQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900031054 (595h6m37.771114089s) [noise=true , address=145.79.227.18:4006 , piece_id=MLZZKCQFU4ZBX4J5FUZHJ26CWSOMSJJHOSDQLGB37GWVHYHXNDUA, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900277222 (595h6m37.771360258s) [noise=true , address=45.91.116.4:4937 , piece_id=IALTP3UKI4PULI2VVKPPTYY5Z2CGWZW3C5V3TYJC3I7G7LUZLM5A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900348268 (595h6m37.771431313s) [noise=true , address=145.79.227.34:4905 , piece_id=DFIORHRM57H5XRZGVIQYL3V4E4AGCWM7C7X2P7QZTWNW6GVQZVLQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900602161 (595h6m37.771685206s) [noise=true , address=145.79.227.7:4295 , piece_id=4RQS6473BFZSNAJD2VVEJNY2D52CVTLZKQ4MBEEJM53AUOL5TRSQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.900657366 (595h6m37.771740401s) [noise=true , address=145.79.227.40:4050 , piece_id=YNGUQV7LIGRE2UZPSZXWHTAMZJE4GY2Y6WRKNHJ2LZWDWKEKLARQ, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:37.901035526 (595h6m37.772118571s) [noise=true , address=45.91.116.7:4012 , piece_id=T72JNTV4TUB23CC5XRPYWJZ5UCRE77WJHGAYM4SK2BAFWTDADY2A, seg_pos={0 0}, upload=521677]: Uploading piece... Dec 30 10:35:39.604350149 (595h6m39.475433194s) [upload=521677]: Next returned nil segment Dec 30 10:35:39.604382581 (595h6m39.475465616s) [upload=521677]: Waiting for error group managing segments... Dec 30 10:35:39.877515232 (595h6m39.748598267s) [noise=true , address=145.79.227.12:4765 , piece_id=HXLI3IRM2WLLXQZZZJDE7XEVDSMEXIAXJHPSDF3XI6AVE75E3SVA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.882503961 (595h6m39.753586996s) [noise=true , address=145.79.227.15:4873 , piece_id=V2LX57CMLFCHSI74REOYCTBYV2XASSNJ3MFSURWZHNDPH2GBLKYA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.888169319 (595h6m39.759252355s) [noise=true , address=145.79.227.19:4079 , piece_id=NHL2UHODXNQOKXPVQMT3DDFGLE4ON3EJZCJW2HSXCVQKYM36XWDA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.888523183 (595h6m39.759606228s) [noise=true , address=145.79.227.22:4510 , piece_id=NPW6KWE3SIYWVUN4WSU5BO2BTEF3S5CYSTPDBBLG52Y6B4RF72QQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.901555459 (595h6m39.772638505s) [noise=true , address=145.79.227.36:4519 , piece_id=274CAK6P7CEUKT6AZGS2IXVTPZVLVFDJ5ZSW6I5J5PCXZLVLCLAQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.904746936 (595h6m39.775829981s) [noise=true , address=145.79.227.51:4409 , piece_id=CTBBMYJHQR6QKV234FMEWOXHOSDZFJCOBGIRBHTHQHCG63UO754A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.910736492 (595h6m39.781819537s) [noise=true , address=145.79.227.7:4295 , piece_id=4RQS6473BFZSNAJD2VVEJNY2D52CVTLZKQ4MBEEJM53AUOL5TRSQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.974959302 (595h6m39.846042337s) [noise=true , address=145.79.227.5:4221 , piece_id=A5U5FCSJB3JXSRALH2GJITQFWMKRP23PHDXI4D6FZMJPYTWGTALQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.981452888 (595h6m39.852535923s) [noise=true , address=145.79.227.18:4006 , piece_id=MLZZKCQFU4ZBX4J5FUZHJ26CWSOMSJJHOSDQLGB37GWVHYHXNDUA, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:39.982670747 (595h6m39.853753782s) [noise=true , address=145.79.227.28:4726 , piece_id=USLT7VCJXW3XIUBKJIWEZXDQW3UCNU7OBJE5W5FA656CV7Z7RBJQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.209859583 (595h6m40.080942618s) [noise=true , address=45.91.116.13:4153 , piece_id=3PAEOVMQUSNSDMHYIHZEIHKRUGCQF3XI3CXYWNYH4MQJX7W3UG3A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.287338457 (595h6m40.158421492s) [noise=true , address=45.91.116.4:4937 , piece_id=IALTP3UKI4PULI2VVKPPTYY5Z2CGWZW3C5V3TYJC3I7G7LUZLM5A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.500418026 (595h6m40.371501071s) [noise=true , address=77.90.31.13:4626 , piece_id=M7X6D4CJSGVAKXBETA325RFNF7766XPWVVIBUWDTCKPXUW2KU6DQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: <nil> Dec 30 10:35:40.500440168 (595h6m40.371523203s) [seg_pos={0 0}, upload=521677]: Segment reached optimal threshold of 13 pieces. Dec 30 10:35:40.500702026 (595h6m40.371785092s) [noise=true , address=145.79.227.34:4905 , piece_id=DFIORHRM57H5XRZGVIQYL3V4E4AGCWM7C7X2P7QZTWNW6GVQZVLQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 30 10:35:40.500712136 (595h6m40.371795181s) [noise=true , address=145.79.227.34:4905 , piece_id=DFIORHRM57H5XRZGVIQYL3V4E4AGCWM7C7X2P7QZTWNW6GVQZVLQ, seg_pos={0 0}, upload=521677]: Long tail cancellation due to optimal threshold Dec 30 10:35:40.500795284 (595h6m40.371878329s) [noise=true , address=145.79.227.40:4050 , piece_id=YNGUQV7LIGRE2UZPSZXWHTAMZJE4GY2Y6WRKNHJ2LZWDWKEKLARQ, seg_pos={0 0}, upload=521677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 30 10:35:40.500824300 (595h6m40.371907345s) [noise=true , address=145.79.227.40:4050 , piece_id=YNGUQV7LIGRE2UZPSZXWHTAMZJE4GY2Y6WRKNHJ2LZWDWKEKLARQ, seg_pos={0 0}, upload=521677]: Long tail cancellation due to optimal threshold Dec 30 10:35:40.501039109 (595h6m40.372122144s) [noise=true , address=45.91.116.7:4012 , piece_id=T72JNTV4TUB23CC5XRPYWJZ5UCRE77WJHGAYM4SK2BAFWTDADY2A, seg_pos={0 0}, upload=521677]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 30 10:35:40.501047384 (595h6m40.372130419s) [noise=true , address=45.91.116.7:4012 , piece_id=T72JNTV4TUB23CC5XRPYWJZ5UCRE77WJHGAYM4SK2BAFWTDADY2A, seg_pos={0 0}, upload=521677]: Long tail cancellation due to optimal threshold Dec 30 10:35:40.501068395 (595h6m40.37215144s) [seg_pos={0 0}, upload=521677]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:35:40.501138688 (595h6m40.372221733s) [upload=521677]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:35:40.501143888 (595h6m40.372226933s) [upload=521677]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:35:40.800282569 (595h6m40.671365605s) [upload=521677]: Done uploading segments... Dec 30 10:35:40.800309010 (595h6m40.671392045s) [upload=521677]: Upload finished. err: <nil> Duration: 30.003874188s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X...............................X....X.........X..........] duration: 29.235394129s trace: https://jaeger.dev.storj.io/trace/1dc8263b2e55d7dc FAIL (2025-12-30 10:35:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311300688 (594h58m28.182383733s) [noise=true , address=145.79.227.43:4179 , piece_id=CQ32O4X3PKQN37OWC3WU5Z5FF7CCAIOAWLUISOXJTTIXTSMKYNQA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311642950 (594h58m28.182725995s) [noise=true , address=145.79.227.30:4783 , piece_id=UIXZYYXWC7KJZUV3BNZAIUSZNDEYA5RFFLKMXWF3XLNKJF5K2A6Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311712022 (594h58m28.182795057s) [noise=true , address=145.79.227.35:4492 , piece_id=GM7TCP65VSFAGWDCRG4VKWAPJHU5HGZFZTVAD4XM6PJFMLCC4NRA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312038483 (594h58m28.183121528s) [noise=true , address=77.90.31.17:4798 , piece_id=HT7E2VVRR2IURW5OIXQ4BLR55QI2BEKJG3VIRG4CABVPCUZRIGWA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312100291 (594h58m28.183183326s) [noise=true , address=77.90.31.19:4859 , piece_id=HEZMENV33Y6A6MMMMDIUECW44662O24WTIG2RWCWZL3IGNKWCH2A, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312494201 (594h58m28.183577246s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312500252 (594h58m28.183583287s) [noise=true , address=145.79.227.49:4359 , piece_id=RGCUJ32JS6NXVVLH7HWUD5IZNV3HZPRRVIHEBJEII2JPEENVIKZA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312899192 (594h58m28.183982227s) [noise=true , address=145.79.227.18:4037 , piece_id=IQZILX3EVK57WA5QGC4VGRYTYWQX37PLIASYIYXIHWBWKGTSEESA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313265008 (594h58m28.184348053s) [noise=true , address=145.79.227.6:4267 , piece_id=ZE5IVJYY2XEW3QOZBBE7LYICAALYG3DZHU6K7UDYUGBHRTWPQYPA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313640203 (594h58m28.184723248s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313904957 (594h58m28.184988012s) [noise=true , address=45.91.116.14:4213 , piece_id=BUTDQ3PI56Q7ZHNYHCEMWPHJUI42LXBR42ZGF4VKMDXJ47BSPU3Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314038000 (594h58m28.185121045s) [noise=true , address=77.90.31.12:4603 , piece_id=QQ5V6RXCHOHVL2QTWLLRFPJBAQQ5KW47T2N44EBMIUW747Q4ZOAA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314410920 (594h58m28.185493965s) [noise=true , address=145.79.227.44:4219 , piece_id=NSHMZDOI264JBC5ML4GRTH66FKT76C5QQ534UA4QDFEIG5P6CHDQ, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314664292 (594h58m28.185747338s) [noise=true , address=145.79.227.40:4062 , piece_id=RTSWJJIEADDCF5Z4JMT4NERSF7EXCWP7QZKDN4KIWEX54UDLNZEA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.315164305 (594h58m28.18624734s) [noise=true , address=77.90.31.10:4540 , piece_id=5LXEPFE5MNCGS6KJK4RP5Q5MV4M3TQMVR5FK4DYTYY4MKTFXTP2Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:29.778852769 (594h58m29.649935804s) [upload=521557]: Next returned nil segment Dec 30 10:27:29.778904818 (594h58m29.649987863s) [upload=521557]: Waiting for error group managing segments... Dec 30 10:27:30.177976438 (594h58m30.049059474s) [noise=true , address=145.79.227.18:4037 , piece_id=IQZILX3EVK57WA5QGC4VGRYTYWQX37PLIASYIYXIHWBWKGTSEESA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.239401413 (594h58m30.110484448s) [noise=true , address=145.79.227.6:4267 , piece_id=ZE5IVJYY2XEW3QOZBBE7LYICAALYG3DZHU6K7UDYUGBHRTWPQYPA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.247231673 (594h58m30.118314708s) [noise=true , address=145.79.227.35:4492 , piece_id=GM7TCP65VSFAGWDCRG4VKWAPJHU5HGZFZTVAD4XM6PJFMLCC4NRA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.364460186 (594h58m30.235543221s) [noise=true , address=77.90.31.17:4798 , piece_id=HT7E2VVRR2IURW5OIXQ4BLR55QI2BEKJG3VIRG4CABVPCUZRIGWA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.415614833 (594h58m30.286697868s) [noise=true , address=77.90.31.12:4603 , piece_id=QQ5V6RXCHOHVL2QTWLLRFPJBAQQ5KW47T2N44EBMIUW747Q4ZOAA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.444886882 (594h58m30.315969917s) [noise=true , address=77.90.31.10:4540 , piece_id=5LXEPFE5MNCGS6KJK4RP5Q5MV4M3TQMVR5FK4DYTYY4MKTFXTP2Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.512047427 (594h58m30.383130462s) [noise=true , address=45.91.116.14:4213 , piece_id=BUTDQ3PI56Q7ZHNYHCEMWPHJUI42LXBR42ZGF4VKMDXJ47BSPU3Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.536847771 (594h58m30.407930806s) [noise=true , address=77.90.31.19:4859 , piece_id=HEZMENV33Y6A6MMMMDIUECW44662O24WTIG2RWCWZL3IGNKWCH2A, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.680580563 (594h58m30.551663608s) [noise=true , address=45.91.116.12:4253 , piece_id=NPLIO7NLEIRTQAWDQ7WZVZE5ZSRLDPDNM34M6SEI53G5XEHB3LJA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.065501378 (594h58m30.936584414s) [noise=true , address=145.79.227.49:4359 , piece_id=RGCUJ32JS6NXVVLH7HWUD5IZNV3HZPRRVIHEBJEII2JPEENVIKZA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.104555514 (594h58m30.975638559s) [noise=true , address=145.79.227.44:4219 , piece_id=NSHMZDOI264JBC5ML4GRTH66FKT76C5QQ534UA4QDFEIG5P6CHDQ, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.115841635 (594h58m30.98692467s) [noise=true , address=145.79.227.30:4783 , piece_id=UIXZYYXWC7KJZUV3BNZAIUSZNDEYA5RFFLKMXWF3XLNKJF5K2A6Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.364792710 (594h58m31.235875755s) [noise=true , address=145.79.227.40:4062 , piece_id=RTSWJJIEADDCF5Z4JMT4NERSF7EXCWP7QZKDN4KIWEX54UDLNZEA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.374552906 (594h58m31.245635941s) [noise=true , address=145.79.227.43:4179 , piece_id=CQ32O4X3PKQN37OWC3WU5Z5FF7CCAIOAWLUISOXJTTIXTSMKYNQA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.374572984 (594h58m31.245656019s) [seg_pos={0 0}, upload=521557]: Segment reached optimal threshold of 14 pieces. Dec 30 10:27:31.374775900 (594h58m31.245858935s) [noise=true , address=45.91.116.10:4115 , piece_id=DRZ2YOFTPSE7SQD6MJ6YQWRQN6MDOLHHP7LWJMKMCPHO37A72C4A, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1arcuTW5rwnvfiqi2H7dL3oL6jKBkDeGq6DmgRGpS9Vs9BCHmf): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.374782553 (594h58m31.245865598s) [noise=true , address=45.91.116.10:4115 , piece_id=DRZ2YOFTPSE7SQD6MJ6YQWRQN6MDOLHHP7LWJMKMCPHO37A72C4A, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.374972405 (594h58m31.24605544s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.374978286 (594h58m31.246061321s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375019294 (594h58m31.246102349s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.375038921 (594h58m31.246121966s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375091411 (594h58m31.246174446s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE): protocol: write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection; write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection Dec 30 10:27:31.375099606 (594h58m31.246182641s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375195900 (594h58m31.246278935s) [seg_pos={0 0}, upload=521557]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:27:31.375261705 (594h58m31.24634474s) [upload=521557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:27:31.375266604 (594h58m31.246349639s) [upload=521557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:27:31.664767884 (594h58m31.53585099s) [upload=521557]: Done uploading segments... Dec 30 10:27:31.664793884 (594h58m31.535876929s) [upload=521557]: Upload finished. err: <nil> Duration: 29.235412574s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [...........XX.X.XX.X.X.X.XX.X.....XX..........XX...X.X..XX.X] duration: 30.003126362s trace: https://jaeger.dev.storj.io/trace/2a3811d1a3725b7e FAIL (2025-12-30 10:35:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 10:35:07.401597972 (595h6m7.272681017s) [upload=521670]: Starting upload Dec 30 10:35:07.401677043 (595h6m7.272760088s) [upload=521670]: Done starting upload Dec 30 10:35:07.401814234 (595h6m7.272897269s) [upload=521670]: Uploading segments... Dec 30 10:35:07.402400861 (595h6m7.273483906s) [upload=521670]: Got next segment. Inline: false Dec 30 10:35:07.402496143 (595h6m7.273579188s) [upload=521670]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:35:07.402503687 (595h6m7.273586732s) [upload=521670]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:35:07.643056952 (595h6m7.514140007s) [seg_pos={0 0}, upload=521670]: Begin upload segment... Dec 30 10:35:07.643107698 (595h6m7.514190743s) [seg_pos={0 0}, upload=521670]: Done begin upload segment. Dec 30 10:35:07.643135371 (595h6m7.514218416s) [noise=true , address=145.79.227.38:4977 , piece_id=BSJSL2FFQKQ5LIXIJRNROF5KNDZFKDJX2QUDX5GBMXDROTNF5LVA, seg_pos={0 0}, upload=521670]: Uploading piece... Dec 30 10:35:07.643340942 (595h6m7.514423987s) [noise=true , address=145.79.227.20:4437 , piece_id=EPJWVCTEN7GTQHXOREQL2LK2RQEPREROD6ASTOIZYAYRVB5SF4AQ, seg_pos={0 0}, upload=521670]: Uploading piece... Dec 30 10:35:07.644338121 (595h6m7.515421197s) [noise=true , address=45.91.116.12:4225 , piece_id=NFKK6TOCL2N6CYERIVVREUM6TVQILL6D3OUQLV77AZ23437J242Q, seg_pos={0 0}, upload=521670]: Uploading piece... Dec 30 10:35:07.644595090 (595h6m7.515678135s) [noise=true , address=145.79.227.51:4409 , piece_id=4XAWZYFGUTE5BLMRBCUUCX4T6NFOLM2ZVRKTLRVAQM33WIXVIMIA, seg_pos={0 0}, upload=521670]: Uploading piece... Dec 30 10:35:07.645344327 (595h6m7.516427373s) [noise=true , address=77.90.31.14:4663 , piece_id=P46TTLLZUZY5XHNG6IYOXKOSEQGS5SSSIJ7BRSBFFRDQR7IS2HXQ, seg_pos={0 0}, upload=521670]: Uploading piece... Dec 30 10:35:07.645926686 (595h6m7.517009731s) [noise=true , address=77.90.31.18:4808 , piece_id=NWN4N76SWOVYVVZZLJ5QVGZIGYPE6VXTHPJRFJ6MMQLQWRV2R7QQ, seg_pos={0 0}, upload=521670]: Uploading piece... Dec 30 10:35:07.646303002 (595h6m7.517386047s) [noise=true , address=145.79.227.5:4221 , piece_id=AQM3BEWSSLIY4PHFQRATWCNFFQ3YGNWN7KD4337EL3STJU4ZTOZA, seg_pos={0 0}, upload=521670]: Uploading piece... Dec 30 10:35:07.646423272 (595h6m7.517506317s) [noise=true , address=77.90.31.13:4629 , piece_id=FIFPWFI5QJ7JGKRLYLOKIZKEUSJ4JLN7O46B5JQNMIALU6VSTLPQ, seg_pos={0 0}, upload=521670]: Uploading piece... Dec 30 10:35:07.647652432 (595h6m7.518735477s) [noise=true , address=145.79.227.4:4185 , piece_id=NZ2XMWT2CZ6WN5JJTLBZIEQ3SJUSL6RNW5B5E6P3IOVAUY3GXYIQ, seg_pos={0 0}, upload=521670]: Uploading piece... Dec 30 10:35:07.647739368 (595h6m7.518822403s) [noise=true , address=45.91.116.13:4155 , piece_id=HG7MRXKEDK5QZMX5SXDWE6W4QROLJHKBBPYWM7TJ2EM4CLV4SPTA, seg_pos={0 0}, upload=521670]: Uploading piece... Dec 30 10:35:07.648272473 (595h6m7.519355518s) [noise=true , address=45.91.116.4:4940 , piece_id=32HYMNGN3M6N3S5OR5KYCKSICDTJBHI3A2ZKU23VMK5FEENQFMOA, seg_pos={0 0}, upload=521670]: Uploading piece... Dec 30 10:35:07.649215919 (595h6m7.520298965s) [noise=true , address=145.79.227.2:4113 , piece_id=FYB7VMJD2SECODX7QKGKYA6MBTKY7QKAKNHZY4AUYCPJEGFC6ZFQ, seg_pos={0 0}, upload=521670]: Uploading piece... Dec 30 10:35:07.650358615 (595h6m7.521441691s) [noise=true , address=145.79.227.52:4445 , piece_id=SSTFC2ZRIFJYRJZNPR77CJU53LUKIL3MCZKWIAIWYXVF6SGSE2QQ, seg_pos={0 0}, upload=521670]: Uploading piece... Dec 30 10:35:07.650644339 (595h6m7.521727384s) [noise=true , address=45.91.116.5:4298 , piece_id=VFZKD534C3QV3G4SHRJQRQS4LIYKRQZOB5FNGZPDBDMHUZQEAPNA, seg_pos={0 0}, upload=521670]: Uploading piece... Dec 30 10:35:07.652019447 (595h6m7.523102492s) [noise=true , address=145.79.227.9:4367 , piece_id=HDINFBVUOVYJQ2PZHUNNHNPSEGC7RAROBTPN2STYC5H624FSDQEQ, seg_pos={0 0}, upload=521670]: Uploading piece... Dec 30 10:35:07.652880487 (595h6m7.523963522s) [noise=true , address=145.79.227.8:4330 , piece_id=33L3ZU4TT5UMAHDYDJTVKDNYCXFZJ5X34YKZUUANDFWCM24X4CNA, seg_pos={0 0}, upload=521670]: Uploading piece... Dec 30 10:35:09.354201392 (595h6m9.225284437s) [upload=521670]: Next returned nil segment Dec 30 10:35:09.354220950 (595h6m9.225304005s) [upload=521670]: Waiting for error group managing segments... Dec 30 10:35:09.638205465 (595h6m9.50928851s) [noise=true , address=145.79.227.5:4221 , piece_id=AQM3BEWSSLIY4PHFQRATWCNFFQ3YGNWN7KD4337EL3STJU4ZTOZA, seg_pos={0 0}, upload=521670]: Done uploading piece. err: <nil> Dec 30 10:35:09.650305656 (595h6m9.521388701s) [noise=true , address=145.79.227.8:4330 , piece_id=33L3ZU4TT5UMAHDYDJTVKDNYCXFZJ5X34YKZUUANDFWCM24X4CNA, seg_pos={0 0}, upload=521670]: Done uploading piece. err: <nil> Dec 30 10:35:09.661884414 (595h6m9.532967459s) [noise=true , address=145.79.227.52:4445 , piece_id=SSTFC2ZRIFJYRJZNPR77CJU53LUKIL3MCZKWIAIWYXVF6SGSE2QQ, seg_pos={0 0}, upload=521670]: Done uploading piece. err: <nil> Dec 30 10:35:09.664089533 (595h6m9.535172568s) [noise=true , address=145.79.227.9:4367 , piece_id=HDINFBVUOVYJQ2PZHUNNHNPSEGC7RAROBTPN2STYC5H624FSDQEQ, seg_pos={0 0}, upload=521670]: Done uploading piece. err: <nil> Dec 30 10:35:09.720615948 (595h6m9.591698993s) [noise=true , address=145.79.227.2:4113 , piece_id=FYB7VMJD2SECODX7QKGKYA6MBTKY7QKAKNHZY4AUYCPJEGFC6ZFQ, seg_pos={0 0}, upload=521670]: Done uploading piece. err: <nil> Dec 30 10:35:09.725450464 (595h6m9.596533499s) [noise=true , address=145.79.227.38:4977 , piece_id=BSJSL2FFQKQ5LIXIJRNROF5KNDZFKDJX2QUDX5GBMXDROTNF5LVA, seg_pos={0 0}, upload=521670]: Done uploading piece. err: <nil> Dec 30 10:35:09.727740775 (595h6m9.59882382s) [noise=true , address=145.79.227.4:4185 , piece_id=NZ2XMWT2CZ6WN5JJTLBZIEQ3SJUSL6RNW5B5E6P3IOVAUY3GXYIQ, seg_pos={0 0}, upload=521670]: Done uploading piece. err: <nil> Dec 30 10:35:09.733169132 (595h6m9.604252187s) [noise=true , address=145.79.227.20:4437 , piece_id=EPJWVCTEN7GTQHXOREQL2LK2RQEPREROD6ASTOIZYAYRVB5SF4AQ, seg_pos={0 0}, upload=521670]: Done uploading piece. err: <nil> Dec 30 10:35:09.801171080 (595h6m9.672254125s) [noise=true , address=145.79.227.51:4409 , piece_id=4XAWZYFGUTE5BLMRBCUUCX4T6NFOLM2ZVRKTLRVAQM33WIXVIMIA, seg_pos={0 0}, upload=521670]: Done uploading piece. err: <nil> Dec 30 10:35:09.899803443 (595h6m9.770886478s) [noise=true , address=77.90.31.14:4663 , piece_id=P46TTLLZUZY5XHNG6IYOXKOSEQGS5SSSIJ7BRSBFFRDQR7IS2HXQ, seg_pos={0 0}, upload=521670]: Done uploading piece. err: <nil> Dec 30 10:35:09.934384350 (595h6m9.805467395s) [noise=true , address=77.90.31.13:4629 , piece_id=FIFPWFI5QJ7JGKRLYLOKIZKEUSJ4JLN7O46B5JQNMIALU6VSTLPQ, seg_pos={0 0}, upload=521670]: Done uploading piece. err: <nil> Dec 30 10:35:09.977267417 (595h6m9.848350452s) [noise=true , address=77.90.31.18:4808 , piece_id=NWN4N76SWOVYVVZZLJ5QVGZIGYPE6VXTHPJRFJ6MMQLQWRV2R7QQ, seg_pos={0 0}, upload=521670]: Done uploading piece. err: <nil> Dec 30 10:35:10.075720378 (595h6m9.946803423s) [noise=true , address=45.91.116.13:4155 , piece_id=HG7MRXKEDK5QZMX5SXDWE6W4QROLJHKBBPYWM7TJ2EM4CLV4SPTA, seg_pos={0 0}, upload=521670]: Done uploading piece. err: <nil> Dec 30 10:35:10.075745116 (595h6m9.946828161s) [seg_pos={0 0}, upload=521670]: Segment reached optimal threshold of 13 pieces. Dec 30 10:35:10.075970023 (595h6m9.947053058s) [noise=true , address=45.91.116.5:4298 , piece_id=VFZKD534C3QV3G4SHRJQRQS4LIYKRQZOB5FNGZPDBDMHUZQEAPNA, seg_pos={0 0}, upload=521670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 30 10:35:10.075977068 (595h6m9.947060103s) [noise=true , address=45.91.116.5:4298 , piece_id=VFZKD534C3QV3G4SHRJQRQS4LIYKRQZOB5FNGZPDBDMHUZQEAPNA, seg_pos={0 0}, upload=521670]: Long tail cancellation due to optimal threshold Dec 30 10:35:10.076080334 (595h6m9.947163379s) [noise=true , address=45.91.116.4:4940 , piece_id=32HYMNGN3M6N3S5OR5KYCKSICDTJBHI3A2ZKU23VMK5FEENQFMOA, seg_pos={0 0}, upload=521670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 30 10:35:10.076088850 (595h6m9.947171885s) [noise=true , address=45.91.116.4:4940 , piece_id=32HYMNGN3M6N3S5OR5KYCKSICDTJBHI3A2ZKU23VMK5FEENQFMOA, seg_pos={0 0}, upload=521670]: Long tail cancellation due to optimal threshold Dec 30 10:35:10.076197106 (595h6m9.947280151s) [noise=true , address=45.91.116.12:4225 , piece_id=NFKK6TOCL2N6CYERIVVREUM6TVQILL6D3OUQLV77AZ23437J242Q, seg_pos={0 0}, upload=521670]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 30 10:35:10.076203318 (595h6m9.947286353s) [noise=true , address=45.91.116.12:4225 , piece_id=NFKK6TOCL2N6CYERIVVREUM6TVQILL6D3OUQLV77AZ23437J242Q, seg_pos={0 0}, upload=521670]: Long tail cancellation due to optimal threshold Dec 30 10:35:10.076247602 (595h6m9.947330647s) [seg_pos={0 0}, upload=521670]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:35:10.076319379 (595h6m9.947402414s) [upload=521670]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:35:10.076323758 (595h6m9.947406793s) [upload=521670]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:35:10.364954520 (595h6m10.236037555s) [upload=521670]: Done uploading segments... Dec 30 10:35:10.365032639 (595h6m10.236115704s) [upload=521670]: Upload finished. err: <nil> Duration: 30.003151611s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [......X......X...X...X........X.....X.X............X......X.] duration: 5.000707654s trace: https://jaeger.dev.storj.io/trace/1efed4bb1724f3ba FAIL (2025-12-30 10:35:33 UTC) orbiter warning: context deadline exceeded Duration: 5.000745866s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X...............................X....X.........X.........] duration: 25.060360902s trace: https://jaeger.dev.storj.io/trace/1dc8263b2e55d7dc FAIL (2025-12-30 10:35:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311300688 (594h58m28.182383733s) [noise=true , address=145.79.227.43:4179 , piece_id=CQ32O4X3PKQN37OWC3WU5Z5FF7CCAIOAWLUISOXJTTIXTSMKYNQA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311642950 (594h58m28.182725995s) [noise=true , address=145.79.227.30:4783 , piece_id=UIXZYYXWC7KJZUV3BNZAIUSZNDEYA5RFFLKMXWF3XLNKJF5K2A6Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311712022 (594h58m28.182795057s) [noise=true , address=145.79.227.35:4492 , piece_id=GM7TCP65VSFAGWDCRG4VKWAPJHU5HGZFZTVAD4XM6PJFMLCC4NRA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312038483 (594h58m28.183121528s) [noise=true , address=77.90.31.17:4798 , piece_id=HT7E2VVRR2IURW5OIXQ4BLR55QI2BEKJG3VIRG4CABVPCUZRIGWA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312100291 (594h58m28.183183326s) [noise=true , address=77.90.31.19:4859 , piece_id=HEZMENV33Y6A6MMMMDIUECW44662O24WTIG2RWCWZL3IGNKWCH2A, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312494201 (594h58m28.183577246s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312500252 (594h58m28.183583287s) [noise=true , address=145.79.227.49:4359 , piece_id=RGCUJ32JS6NXVVLH7HWUD5IZNV3HZPRRVIHEBJEII2JPEENVIKZA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312899192 (594h58m28.183982227s) [noise=true , address=145.79.227.18:4037 , piece_id=IQZILX3EVK57WA5QGC4VGRYTYWQX37PLIASYIYXIHWBWKGTSEESA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313265008 (594h58m28.184348053s) [noise=true , address=145.79.227.6:4267 , piece_id=ZE5IVJYY2XEW3QOZBBE7LYICAALYG3DZHU6K7UDYUGBHRTWPQYPA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313640203 (594h58m28.184723248s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313904957 (594h58m28.184988012s) [noise=true , address=45.91.116.14:4213 , piece_id=BUTDQ3PI56Q7ZHNYHCEMWPHJUI42LXBR42ZGF4VKMDXJ47BSPU3Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314038000 (594h58m28.185121045s) [noise=true , address=77.90.31.12:4603 , piece_id=QQ5V6RXCHOHVL2QTWLLRFPJBAQQ5KW47T2N44EBMIUW747Q4ZOAA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314410920 (594h58m28.185493965s) [noise=true , address=145.79.227.44:4219 , piece_id=NSHMZDOI264JBC5ML4GRTH66FKT76C5QQ534UA4QDFEIG5P6CHDQ, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314664292 (594h58m28.185747338s) [noise=true , address=145.79.227.40:4062 , piece_id=RTSWJJIEADDCF5Z4JMT4NERSF7EXCWP7QZKDN4KIWEX54UDLNZEA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.315164305 (594h58m28.18624734s) [noise=true , address=77.90.31.10:4540 , piece_id=5LXEPFE5MNCGS6KJK4RP5Q5MV4M3TQMVR5FK4DYTYY4MKTFXTP2Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:29.778852769 (594h58m29.649935804s) [upload=521557]: Next returned nil segment Dec 30 10:27:29.778904818 (594h58m29.649987863s) [upload=521557]: Waiting for error group managing segments... Dec 30 10:27:30.177976438 (594h58m30.049059474s) [noise=true , address=145.79.227.18:4037 , piece_id=IQZILX3EVK57WA5QGC4VGRYTYWQX37PLIASYIYXIHWBWKGTSEESA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.239401413 (594h58m30.110484448s) [noise=true , address=145.79.227.6:4267 , piece_id=ZE5IVJYY2XEW3QOZBBE7LYICAALYG3DZHU6K7UDYUGBHRTWPQYPA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.247231673 (594h58m30.118314708s) [noise=true , address=145.79.227.35:4492 , piece_id=GM7TCP65VSFAGWDCRG4VKWAPJHU5HGZFZTVAD4XM6PJFMLCC4NRA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.364460186 (594h58m30.235543221s) [noise=true , address=77.90.31.17:4798 , piece_id=HT7E2VVRR2IURW5OIXQ4BLR55QI2BEKJG3VIRG4CABVPCUZRIGWA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.415614833 (594h58m30.286697868s) [noise=true , address=77.90.31.12:4603 , piece_id=QQ5V6RXCHOHVL2QTWLLRFPJBAQQ5KW47T2N44EBMIUW747Q4ZOAA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.444886882 (594h58m30.315969917s) [noise=true , address=77.90.31.10:4540 , piece_id=5LXEPFE5MNCGS6KJK4RP5Q5MV4M3TQMVR5FK4DYTYY4MKTFXTP2Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.512047427 (594h58m30.383130462s) [noise=true , address=45.91.116.14:4213 , piece_id=BUTDQ3PI56Q7ZHNYHCEMWPHJUI42LXBR42ZGF4VKMDXJ47BSPU3Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.536847771 (594h58m30.407930806s) [noise=true , address=77.90.31.19:4859 , piece_id=HEZMENV33Y6A6MMMMDIUECW44662O24WTIG2RWCWZL3IGNKWCH2A, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.680580563 (594h58m30.551663608s) [noise=true , address=45.91.116.12:4253 , piece_id=NPLIO7NLEIRTQAWDQ7WZVZE5ZSRLDPDNM34M6SEI53G5XEHB3LJA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.065501378 (594h58m30.936584414s) [noise=true , address=145.79.227.49:4359 , piece_id=RGCUJ32JS6NXVVLH7HWUD5IZNV3HZPRRVIHEBJEII2JPEENVIKZA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.104555514 (594h58m30.975638559s) [noise=true , address=145.79.227.44:4219 , piece_id=NSHMZDOI264JBC5ML4GRTH66FKT76C5QQ534UA4QDFEIG5P6CHDQ, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.115841635 (594h58m30.98692467s) [noise=true , address=145.79.227.30:4783 , piece_id=UIXZYYXWC7KJZUV3BNZAIUSZNDEYA5RFFLKMXWF3XLNKJF5K2A6Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.364792710 (594h58m31.235875755s) [noise=true , address=145.79.227.40:4062 , piece_id=RTSWJJIEADDCF5Z4JMT4NERSF7EXCWP7QZKDN4KIWEX54UDLNZEA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.374552906 (594h58m31.245635941s) [noise=true , address=145.79.227.43:4179 , piece_id=CQ32O4X3PKQN37OWC3WU5Z5FF7CCAIOAWLUISOXJTTIXTSMKYNQA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.374572984 (594h58m31.245656019s) [seg_pos={0 0}, upload=521557]: Segment reached optimal threshold of 14 pieces. Dec 30 10:27:31.374775900 (594h58m31.245858935s) [noise=true , address=45.91.116.10:4115 , piece_id=DRZ2YOFTPSE7SQD6MJ6YQWRQN6MDOLHHP7LWJMKMCPHO37A72C4A, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1arcuTW5rwnvfiqi2H7dL3oL6jKBkDeGq6DmgRGpS9Vs9BCHmf): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.374782553 (594h58m31.245865598s) [noise=true , address=45.91.116.10:4115 , piece_id=DRZ2YOFTPSE7SQD6MJ6YQWRQN6MDOLHHP7LWJMKMCPHO37A72C4A, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.374972405 (594h58m31.24605544s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.374978286 (594h58m31.246061321s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375019294 (594h58m31.246102349s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.375038921 (594h58m31.246121966s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375091411 (594h58m31.246174446s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE): protocol: write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection; write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection Dec 30 10:27:31.375099606 (594h58m31.246182641s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375195900 (594h58m31.246278935s) [seg_pos={0 0}, upload=521557]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:27:31.375261705 (594h58m31.24634474s) [upload=521557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:27:31.375266604 (594h58m31.246349639s) [upload=521557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:27:31.664767884 (594h58m31.53585099s) [upload=521557]: Done uploading segments... Dec 30 10:27:31.664793884 (594h58m31.535876929s) [upload=521557]: Upload finished. err: <nil> Duration: 25.060389126s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [X......X......X...X...X........X.....X.X............X......X] duration: 5.001498488s trace: https://jaeger.dev.storj.io/trace/1efed4bb1724f3ba FAIL (2025-12-30 10:34:30 UTC) orbiter warning: context deadline exceeded Duration: 5.001528984s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [............XX.X.XX.X.X.X.XX.X.....XX..........XX...X.X..XX.] duration: 5.904785168s trace: https://jaeger.dev.storj.io/trace/3e8ecd1cafb0c66c FAIL (2025-12-30 10:34:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: RQUDUJYXWLRBFGYPC2QVVBGUFUDXIVXCTQSTADHMLJ3KHPEV4YKA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: N2HR7RCOJ7U3RLKJQBTDJKSB22NTKQRFNOFUZ2DB726QKOTZ5VZA) context deadline exceeded ======LOGS====== Dec 30 10:33:51.195955555 (595h4m51.0670386s) [upload=521652]: Starting upload Dec 30 10:33:51.196017303 (595h4m51.067100338s) [upload=521652]: Done starting upload Dec 30 10:33:51.196167399 (595h4m51.067250444s) [upload=521652]: Uploading segments... Dec 30 10:33:51.196311253 (595h4m51.067394298s) [upload=521652]: Got next segment. Inline: false Dec 30 10:33:51.196353734 (595h4m51.067436779s) [upload=521652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:33:51.196360657 (595h4m51.067443702s) [upload=521652]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:33:51.441699243 (595h4m51.312782278s) [seg_pos={0 0}, upload=521652]: Begin upload segment... Dec 30 10:33:51.441748868 (595h4m51.312831913s) [seg_pos={0 0}, upload=521652]: Done begin upload segment. Dec 30 10:33:51.441790287 (595h4m51.312873322s) [noise=true , address=145.79.227.39:4015 , piece_id=JZWZABYLYCLRX3KGVMGXXNNTXIMU57D5EYYPKQSCN3ND4B27ULDA, seg_pos={0 0}, upload=521652]: Uploading piece... Dec 30 10:33:51.441833228 (595h4m51.312916273s) [noise=true , address=45.91.116.9:4262 , piece_id=ENXUX62WCLTCIKLRMTAAC3PJL7F2TTLWVUSL2AAVFKB757PQXEUA, seg_pos={0 0}, upload=521652]: Uploading piece... Dec 30 10:33:51.442444723 (595h4m51.313527768s) [noise=true , address=145.79.227.3:4149 , piece_id=FU4GQT7E6W7VBBTUPNRW4I6P7T5XPXGWXEXKPDGIPUK2VJAMCD3Q, seg_pos={0 0}, upload=521652]: Uploading piece... Dec 30 10:33:51.442814867 (595h4m51.313897902s) [noise=true , address=45.91.116.11:4119 , piece_id=RQUDUJYXWLRBFGYPC2QVVBGUFUDXIVXCTQSTADHMLJ3KHPEV4YKA, seg_pos={0 0}, upload=521652]: Uploading piece... Dec 30 10:33:51.442949163 (595h4m51.314032198s) [noise=true , address=145.79.227.38:4979 , piece_id=JLJZK4R3OLCIUSCHUKEHFBP6XJKEWZFU6TNRLV56RN5BGCONP55A, seg_pos={0 0}, upload=521652]: Uploading piece... Dec 30 10:33:51.443925632 (595h4m51.315008677s) [noise=true , address=77.90.31.12:4590 , piece_id=N2HR7RCOJ7U3RLKJQBTDJKSB22NTKQRFNOFUZ2DB726QKOTZ5VZA, seg_pos={0 0}, upload=521652]: Uploading piece... Dec 30 10:33:51.444413561 (595h4m51.315496597s) [noise=true , address=145.79.227.10:4401 , piece_id=4N3ZF3SZF54WLGILM5LG3JTR6AWVROWBL33DKY7R3XMQ2D2SL4OQ, seg_pos={0 0}, upload=521652]: Uploading piece... Dec 30 10:33:51.445324767 (595h4m51.316407802s) [noise=true , address=145.79.227.25:4617 , piece_id=BUL7QARBANFHNLQHXHHYLK4FXA6BZ2ZVGBF4RIKYI3QCEPVDKEIA, seg_pos={0 0}, upload=521652]: Uploading piece... Dec 30 10:33:51.445706144 (595h4m51.316789179s) [noise=true , address=145.79.227.17:4972 , piece_id=3QPUUPSKB3XHKO6POE2LOMGITKPTNMJ27D6H4HIVGH44SQL476WA, seg_pos={0 0}, upload=521652]: Uploading piece... Dec 30 10:33:51.446198461 (595h4m51.317281496s) [noise=true , address=145.79.227.4:4187 , piece_id=ZJF6FSBVCX3Z2EBNGLNQBPWTTBIUNXQMOG7FQNCQUOXG5TEGG6FQ, seg_pos={0 0}, upload=521652]: Uploading piece... Dec 30 10:33:51.446875410 (595h4m51.317958455s) [noise=true , address=145.79.227.37:4566 , piece_id=HUMNI4C76XCN6CNPXHDPUZSEDMZ2JOGTR6DJRPROLOZON5QRM2EQ, seg_pos={0 0}, upload=521652]: Uploading piece... Dec 30 10:33:51.447386523 (595h4m51.318469558s) [noise=true , address=145.79.227.15:4875 , piece_id=TEVKPIIW4OX5YXKKEAHXPHMZMED456DEAF5O7RDE2VE7KWLAMF3Q, seg_pos={0 0}, upload=521652]: Uploading piece... Dec 30 10:33:51.447676275 (595h4m51.31875931s) [noise=true , address=45.91.116.13:4153 , piece_id=P4YOMCC54K74W5C633MC4BBCIMIVO7JPFXAZYZCA4AVK5KESUUBA, seg_pos={0 0}, upload=521652]: Uploading piece... Dec 30 10:33:51.448165817 (595h4m51.319248852s) [noise=true , address=145.79.227.44:4195 , piece_id=2TNEDLZMMZCYDRRISUCY243QYZEOYAT7L2IMZ6WGHV2FYBDCMFFA, seg_pos={0 0}, upload=521652]: Uploading piece... Dec 30 10:33:51.448426514 (595h4m51.319509559s) [noise=true , address=145.79.227.6:4258 , piece_id=X5QQL5MVMCJZMJMHNEZRNEIOI7NO7WCQLXOFOJQ5RVTADE4QEXDQ, seg_pos={0 0}, upload=521652]: Uploading piece... Dec 30 10:33:51.448798431 (595h4m51.319881466s) [noise=true , address=145.79.227.52:4446 , piece_id=J3RNYKVPDVFMVQQRPRYL65XVCIGRRX6MTT2HBQ7LQ5MVRTWKYBYQ, seg_pos={0 0}, upload=521652]: Uploading piece... Dec 30 10:33:53.139669478 (595h4m53.010752523s) [upload=521652]: Next returned nil segment Dec 30 10:33:53.139702461 (595h4m53.010785496s) [upload=521652]: Waiting for error group managing segments... Dec 30 10:33:53.421590725 (595h4m53.29267376s) [noise=true , address=145.79.227.37:4566 , piece_id=HUMNI4C76XCN6CNPXHDPUZSEDMZ2JOGTR6DJRPROLOZON5QRM2EQ, seg_pos={0 0}, upload=521652]: Done uploading piece. err: <nil> Dec 30 10:33:53.425871807 (595h4m53.296954852s) [noise=true , address=145.79.227.17:4972 , piece_id=3QPUUPSKB3XHKO6POE2LOMGITKPTNMJ27D6H4HIVGH44SQL476WA, seg_pos={0 0}, upload=521652]: Done uploading piece. err: <nil> Dec 30 10:33:53.428237572 (595h4m53.299320607s) [noise=true , address=145.79.227.3:4149 , piece_id=FU4GQT7E6W7VBBTUPNRW4I6P7T5XPXGWXEXKPDGIPUK2VJAMCD3Q, seg_pos={0 0}, upload=521652]: Done uploading piece. err: <nil> Dec 30 10:33:53.434511118 (595h4m53.305594153s) [noise=true , address=145.79.227.15:4875 , piece_id=TEVKPIIW4OX5YXKKEAHXPHMZMED456DEAF5O7RDE2VE7KWLAMF3Q, seg_pos={0 0}, upload=521652]: Done uploading piece. err: <nil> Dec 30 10:33:53.436396478 (595h4m53.307479513s) [noise=true , address=145.79.227.10:4401 , piece_id=4N3ZF3SZF54WLGILM5LG3JTR6AWVROWBL33DKY7R3XMQ2D2SL4OQ, seg_pos={0 0}, upload=521652]: Done uploading piece. err: <nil> Dec 30 10:33:53.448966703 (595h4m53.320049738s) [noise=true , address=145.79.227.25:4617 , piece_id=BUL7QARBANFHNLQHXHHYLK4FXA6BZ2ZVGBF4RIKYI3QCEPVDKEIA, seg_pos={0 0}, upload=521652]: Done uploading piece. err: <nil> Dec 30 10:33:53.450425441 (595h4m53.321508476s) [noise=true , address=145.79.227.52:4446 , piece_id=J3RNYKVPDVFMVQQRPRYL65XVCIGRRX6MTT2HBQ7LQ5MVRTWKYBYQ, seg_pos={0 0}, upload=521652]: Done uploading piece. err: <nil> Dec 30 10:33:53.452173199 (595h4m53.323256234s) [noise=true , address=145.79.227.38:4979 , piece_id=JLJZK4R3OLCIUSCHUKEHFBP6XJKEWZFU6TNRLV56RN5BGCONP55A, seg_pos={0 0}, upload=521652]: Done uploading piece. err: <nil> Dec 30 10:33:53.512446810 (595h4m53.383529845s) [noise=true , address=145.79.227.6:4258 , piece_id=X5QQL5MVMCJZMJMHNEZRNEIOI7NO7WCQLXOFOJQ5RVTADE4QEXDQ, seg_pos={0 0}, upload=521652]: Done uploading piece. err: <nil> Dec 30 10:33:53.513188693 (595h4m53.384271738s) [noise=true , address=145.79.227.39:4015 , piece_id=JZWZABYLYCLRX3KGVMGXXNNTXIMU57D5EYYPKQSCN3ND4B27ULDA, seg_pos={0 0}, upload=521652]: Done uploading piece. err: <nil> Dec 30 10:33:53.514273910 (595h4m53.385356955s) [noise=true , address=145.79.227.4:4187 , piece_id=ZJF6FSBVCX3Z2EBNGLNQBPWTTBIUNXQMOG7FQNCQUOXG5TEGG6FQ, seg_pos={0 0}, upload=521652]: Done uploading piece. err: <nil> Dec 30 10:33:53.721726384 (595h4m53.592809429s) [noise=true , address=77.90.31.12:4590 , piece_id=N2HR7RCOJ7U3RLKJQBTDJKSB22NTKQRFNOFUZ2DB726QKOTZ5VZA, seg_pos={0 0}, upload=521652]: Done uploading piece. err: <nil> Dec 30 10:33:54.143436022 (595h4m54.014519057s) [noise=true , address=45.91.116.11:4119 , piece_id=RQUDUJYXWLRBFGYPC2QVVBGUFUDXIVXCTQSTADHMLJ3KHPEV4YKA, seg_pos={0 0}, upload=521652]: Done uploading piece. err: <nil> Dec 30 10:33:54.143456010 (595h4m54.014539055s) [seg_pos={0 0}, upload=521652]: Segment reached optimal threshold of 13 pieces. Dec 30 10:33:54.143657714 (595h4m54.014740759s) [noise=true , address=145.79.227.44:4195 , piece_id=2TNEDLZMMZCYDRRISUCY243QYZEOYAT7L2IMZ6WGHV2FYBDCMFFA, seg_pos={0 0}, upload=521652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 30 10:33:54.143663735 (595h4m54.01474677s) [noise=true , address=145.79.227.44:4195 , piece_id=2TNEDLZMMZCYDRRISUCY243QYZEOYAT7L2IMZ6WGHV2FYBDCMFFA, seg_pos={0 0}, upload=521652]: Long tail cancellation due to optimal threshold Dec 30 10:33:54.143781049 (595h4m54.014864084s) [noise=true , address=45.91.116.9:4262 , piece_id=ENXUX62WCLTCIKLRMTAAC3PJL7F2TTLWVUSL2AAVFKB757PQXEUA, seg_pos={0 0}, upload=521652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:57422->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:57422->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57422->45.91.116.9:4262: use of closed network connection Dec 30 10:33:54.143786239 (595h4m54.014869284s) [noise=true , address=45.91.116.9:4262 , piece_id=ENXUX62WCLTCIKLRMTAAC3PJL7F2TTLWVUSL2AAVFKB757PQXEUA, seg_pos={0 0}, upload=521652]: Long tail cancellation due to optimal threshold Dec 30 10:33:54.143917369 (595h4m54.015000404s) [noise=true , address=45.91.116.13:4153 , piece_id=P4YOMCC54K74W5C633MC4BBCIMIVO7JPFXAZYZCA4AVK5KESUUBA, seg_pos={0 0}, upload=521652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 30 10:33:54.143975308 (595h4m54.015058343s) [noise=true , address=45.91.116.13:4153 , piece_id=P4YOMCC54K74W5C633MC4BBCIMIVO7JPFXAZYZCA4AVK5KESUUBA, seg_pos={0 0}, upload=521652]: Long tail cancellation due to optimal threshold Dec 30 10:33:54.144007961 (595h4m54.015090996s) [seg_pos={0 0}, upload=521652]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:33:54.144182904 (595h4m54.015265939s) [upload=521652]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:33:54.144728573 (595h4m54.015811738s) [upload=521652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:33:54.426965170 (595h4m54.298048215s) [upload=521652]: Done uploading segments... Dec 30 10:33:54.426989787 (595h4m54.298072832s) [upload=521652]: Upload finished. err: <nil> Duration: 5.904800989s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.............XX.X.XX.X.X.X.XX.X.....XX..........XX...X.X..XX] duration: 30.002592151s trace: https://jaeger.dev.storj.io/trace/3e8ecd1cafb0c66c FAIL (2025-12-30 10:34:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, Piece ID: RQUDUJYXWLRBFGYPC2QVVBGUFUDXIVXCTQSTADHMLJ3KHPEV4YKA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: N2HR7RCOJ7U3RLKJQBTDJKSB22NTKQRFNOFUZ2DB726QKOTZ5VZA) context deadline exceeded ======LOGS====== Dec 30 10:33:51.195955555 (595h4m51.0670386s) [upload=521652]: Starting upload Dec 30 10:33:51.196017303 (595h4m51.067100338s) [upload=521652]: Done starting upload Dec 30 10:33:51.196167399 (595h4m51.067250444s) [upload=521652]: Uploading segments... Dec 30 10:33:51.196311253 (595h4m51.067394298s) [upload=521652]: Got next segment. Inline: false Dec 30 10:33:51.196353734 (595h4m51.067436779s) [upload=521652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 10:33:51.196360657 (595h4m51.067443702s) [upload=521652]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 10:33:51.441699243 (595h4m51.312782278s) [seg_pos={0 0}, upload=521652]: Begin upload segment... Dec 30 10:33:51.441748868 (595h4m51.312831913s) [seg_pos={0 0}, upload=521652]: Done begin upload segment. Dec 30 10:33:51.441790287 (595h4m51.312873322s) [noise=true , address=145.79.227.39:4015 , piece_id=JZWZABYLYCLRX3KGVMGXXNNTXIMU57D5EYYPKQSCN3ND4B27ULDA, seg_pos={0 0}, upload=521652]: Uploading piece... Dec 30 10:33:51.441833228 (595h4m51.312916273s) [noise=true , address=45.91.116.9:4262 , piece_id=ENXUX62WCLTCIKLRMTAAC3PJL7F2TTLWVUSL2AAVFKB757PQXEUA, seg_pos={0 0}, upload=521652]: Uploading piece... Dec 30 10:33:51.442444723 (595h4m51.313527768s) [noise=true , address=145.79.227.3:4149 , piece_id=FU4GQT7E6W7VBBTUPNRW4I6P7T5XPXGWXEXKPDGIPUK2VJAMCD3Q, seg_pos={0 0}, upload=521652]: Uploading piece... Dec 30 10:33:51.442814867 (595h4m51.313897902s) [noise=true , address=45.91.116.11:4119 , piece_id=RQUDUJYXWLRBFGYPC2QVVBGUFUDXIVXCTQSTADHMLJ3KHPEV4YKA, seg_pos={0 0}, upload=521652]: Uploading piece... Dec 30 10:33:51.442949163 (595h4m51.314032198s) [noise=true , address=145.79.227.38:4979 , piece_id=JLJZK4R3OLCIUSCHUKEHFBP6XJKEWZFU6TNRLV56RN5BGCONP55A, seg_pos={0 0}, upload=521652]: Uploading piece... Dec 30 10:33:51.443925632 (595h4m51.315008677s) [noise=true , address=77.90.31.12:4590 , piece_id=N2HR7RCOJ7U3RLKJQBTDJKSB22NTKQRFNOFUZ2DB726QKOTZ5VZA, seg_pos={0 0}, upload=521652]: Uploading piece... Dec 30 10:33:51.444413561 (595h4m51.315496597s) [noise=true , address=145.79.227.10:4401 , piece_id=4N3ZF3SZF54WLGILM5LG3JTR6AWVROWBL33DKY7R3XMQ2D2SL4OQ, seg_pos={0 0}, upload=521652]: Uploading piece... Dec 30 10:33:51.445324767 (595h4m51.316407802s) [noise=true , address=145.79.227.25:4617 , piece_id=BUL7QARBANFHNLQHXHHYLK4FXA6BZ2ZVGBF4RIKYI3QCEPVDKEIA, seg_pos={0 0}, upload=521652]: Uploading piece... Dec 30 10:33:51.445706144 (595h4m51.316789179s) [noise=true , address=145.79.227.17:4972 , piece_id=3QPUUPSKB3XHKO6POE2LOMGITKPTNMJ27D6H4HIVGH44SQL476WA, seg_pos={0 0}, upload=521652]: Uploading piece... Dec 30 10:33:51.446198461 (595h4m51.317281496s) [noise=true , address=145.79.227.4:4187 , piece_id=ZJF6FSBVCX3Z2EBNGLNQBPWTTBIUNXQMOG7FQNCQUOXG5TEGG6FQ, seg_pos={0 0}, upload=521652]: Uploading piece... Dec 30 10:33:51.446875410 (595h4m51.317958455s) [noise=true , address=145.79.227.37:4566 , piece_id=HUMNI4C76XCN6CNPXHDPUZSEDMZ2JOGTR6DJRPROLOZON5QRM2EQ, seg_pos={0 0}, upload=521652]: Uploading piece... Dec 30 10:33:51.447386523 (595h4m51.318469558s) [noise=true , address=145.79.227.15:4875 , piece_id=TEVKPIIW4OX5YXKKEAHXPHMZMED456DEAF5O7RDE2VE7KWLAMF3Q, seg_pos={0 0}, upload=521652]: Uploading piece... Dec 30 10:33:51.447676275 (595h4m51.31875931s) [noise=true , address=45.91.116.13:4153 , piece_id=P4YOMCC54K74W5C633MC4BBCIMIVO7JPFXAZYZCA4AVK5KESUUBA, seg_pos={0 0}, upload=521652]: Uploading piece... Dec 30 10:33:51.448165817 (595h4m51.319248852s) [noise=true , address=145.79.227.44:4195 , piece_id=2TNEDLZMMZCYDRRISUCY243QYZEOYAT7L2IMZ6WGHV2FYBDCMFFA, seg_pos={0 0}, upload=521652]: Uploading piece... Dec 30 10:33:51.448426514 (595h4m51.319509559s) [noise=true , address=145.79.227.6:4258 , piece_id=X5QQL5MVMCJZMJMHNEZRNEIOI7NO7WCQLXOFOJQ5RVTADE4QEXDQ, seg_pos={0 0}, upload=521652]: Uploading piece... Dec 30 10:33:51.448798431 (595h4m51.319881466s) [noise=true , address=145.79.227.52:4446 , piece_id=J3RNYKVPDVFMVQQRPRYL65XVCIGRRX6MTT2HBQ7LQ5MVRTWKYBYQ, seg_pos={0 0}, upload=521652]: Uploading piece... Dec 30 10:33:53.139669478 (595h4m53.010752523s) [upload=521652]: Next returned nil segment Dec 30 10:33:53.139702461 (595h4m53.010785496s) [upload=521652]: Waiting for error group managing segments... Dec 30 10:33:53.421590725 (595h4m53.29267376s) [noise=true , address=145.79.227.37:4566 , piece_id=HUMNI4C76XCN6CNPXHDPUZSEDMZ2JOGTR6DJRPROLOZON5QRM2EQ, seg_pos={0 0}, upload=521652]: Done uploading piece. err: <nil> Dec 30 10:33:53.425871807 (595h4m53.296954852s) [noise=true , address=145.79.227.17:4972 , piece_id=3QPUUPSKB3XHKO6POE2LOMGITKPTNMJ27D6H4HIVGH44SQL476WA, seg_pos={0 0}, upload=521652]: Done uploading piece. err: <nil> Dec 30 10:33:53.428237572 (595h4m53.299320607s) [noise=true , address=145.79.227.3:4149 , piece_id=FU4GQT7E6W7VBBTUPNRW4I6P7T5XPXGWXEXKPDGIPUK2VJAMCD3Q, seg_pos={0 0}, upload=521652]: Done uploading piece. err: <nil> Dec 30 10:33:53.434511118 (595h4m53.305594153s) [noise=true , address=145.79.227.15:4875 , piece_id=TEVKPIIW4OX5YXKKEAHXPHMZMED456DEAF5O7RDE2VE7KWLAMF3Q, seg_pos={0 0}, upload=521652]: Done uploading piece. err: <nil> Dec 30 10:33:53.436396478 (595h4m53.307479513s) [noise=true , address=145.79.227.10:4401 , piece_id=4N3ZF3SZF54WLGILM5LG3JTR6AWVROWBL33DKY7R3XMQ2D2SL4OQ, seg_pos={0 0}, upload=521652]: Done uploading piece. err: <nil> Dec 30 10:33:53.448966703 (595h4m53.320049738s) [noise=true , address=145.79.227.25:4617 , piece_id=BUL7QARBANFHNLQHXHHYLK4FXA6BZ2ZVGBF4RIKYI3QCEPVDKEIA, seg_pos={0 0}, upload=521652]: Done uploading piece. err: <nil> Dec 30 10:33:53.450425441 (595h4m53.321508476s) [noise=true , address=145.79.227.52:4446 , piece_id=J3RNYKVPDVFMVQQRPRYL65XVCIGRRX6MTT2HBQ7LQ5MVRTWKYBYQ, seg_pos={0 0}, upload=521652]: Done uploading piece. err: <nil> Dec 30 10:33:53.452173199 (595h4m53.323256234s) [noise=true , address=145.79.227.38:4979 , piece_id=JLJZK4R3OLCIUSCHUKEHFBP6XJKEWZFU6TNRLV56RN5BGCONP55A, seg_pos={0 0}, upload=521652]: Done uploading piece. err: <nil> Dec 30 10:33:53.512446810 (595h4m53.383529845s) [noise=true , address=145.79.227.6:4258 , piece_id=X5QQL5MVMCJZMJMHNEZRNEIOI7NO7WCQLXOFOJQ5RVTADE4QEXDQ, seg_pos={0 0}, upload=521652]: Done uploading piece. err: <nil> Dec 30 10:33:53.513188693 (595h4m53.384271738s) [noise=true , address=145.79.227.39:4015 , piece_id=JZWZABYLYCLRX3KGVMGXXNNTXIMU57D5EYYPKQSCN3ND4B27ULDA, seg_pos={0 0}, upload=521652]: Done uploading piece. err: <nil> Dec 30 10:33:53.514273910 (595h4m53.385356955s) [noise=true , address=145.79.227.4:4187 , piece_id=ZJF6FSBVCX3Z2EBNGLNQBPWTTBIUNXQMOG7FQNCQUOXG5TEGG6FQ, seg_pos={0 0}, upload=521652]: Done uploading piece. err: <nil> Dec 30 10:33:53.721726384 (595h4m53.592809429s) [noise=true , address=77.90.31.12:4590 , piece_id=N2HR7RCOJ7U3RLKJQBTDJKSB22NTKQRFNOFUZ2DB726QKOTZ5VZA, seg_pos={0 0}, upload=521652]: Done uploading piece. err: <nil> Dec 30 10:33:54.143436022 (595h4m54.014519057s) [noise=true , address=45.91.116.11:4119 , piece_id=RQUDUJYXWLRBFGYPC2QVVBGUFUDXIVXCTQSTADHMLJ3KHPEV4YKA, seg_pos={0 0}, upload=521652]: Done uploading piece. err: <nil> Dec 30 10:33:54.143456010 (595h4m54.014539055s) [seg_pos={0 0}, upload=521652]: Segment reached optimal threshold of 13 pieces. Dec 30 10:33:54.143657714 (595h4m54.014740759s) [noise=true , address=145.79.227.44:4195 , piece_id=2TNEDLZMMZCYDRRISUCY243QYZEOYAT7L2IMZ6WGHV2FYBDCMFFA, seg_pos={0 0}, upload=521652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 30 10:33:54.143663735 (595h4m54.01474677s) [noise=true , address=145.79.227.44:4195 , piece_id=2TNEDLZMMZCYDRRISUCY243QYZEOYAT7L2IMZ6WGHV2FYBDCMFFA, seg_pos={0 0}, upload=521652]: Long tail cancellation due to optimal threshold Dec 30 10:33:54.143781049 (595h4m54.014864084s) [noise=true , address=45.91.116.9:4262 , piece_id=ENXUX62WCLTCIKLRMTAAC3PJL7F2TTLWVUSL2AAVFKB757PQXEUA, seg_pos={0 0}, upload=521652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:57422->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:57422->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57422->45.91.116.9:4262: use of closed network connection Dec 30 10:33:54.143786239 (595h4m54.014869284s) [noise=true , address=45.91.116.9:4262 , piece_id=ENXUX62WCLTCIKLRMTAAC3PJL7F2TTLWVUSL2AAVFKB757PQXEUA, seg_pos={0 0}, upload=521652]: Long tail cancellation due to optimal threshold Dec 30 10:33:54.143917369 (595h4m54.015000404s) [noise=true , address=45.91.116.13:4153 , piece_id=P4YOMCC54K74W5C633MC4BBCIMIVO7JPFXAZYZCA4AVK5KESUUBA, seg_pos={0 0}, upload=521652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 30 10:33:54.143975308 (595h4m54.015058343s) [noise=true , address=45.91.116.13:4153 , piece_id=P4YOMCC54K74W5C633MC4BBCIMIVO7JPFXAZYZCA4AVK5KESUUBA, seg_pos={0 0}, upload=521652]: Long tail cancellation due to optimal threshold Dec 30 10:33:54.144007961 (595h4m54.015090996s) [seg_pos={0 0}, upload=521652]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 10:33:54.144182904 (595h4m54.015265939s) [upload=521652]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:33:54.144728573 (595h4m54.015811738s) [upload=521652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:33:54.426965170 (595h4m54.298048215s) [upload=521652]: Done uploading segments... Dec 30 10:33:54.426989787 (595h4m54.298072832s) [upload=521652]: Upload finished. err: <nil> Duration: 30.002605256s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X...............................X....X.........X........] duration: 15.319384124s trace: https://jaeger.dev.storj.io/trace/1dc8263b2e55d7dc FAIL (2025-12-30 10:34:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311300688 (594h58m28.182383733s) [noise=true , address=145.79.227.43:4179 , piece_id=CQ32O4X3PKQN37OWC3WU5Z5FF7CCAIOAWLUISOXJTTIXTSMKYNQA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311642950 (594h58m28.182725995s) [noise=true , address=145.79.227.30:4783 , piece_id=UIXZYYXWC7KJZUV3BNZAIUSZNDEYA5RFFLKMXWF3XLNKJF5K2A6Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.311712022 (594h58m28.182795057s) [noise=true , address=145.79.227.35:4492 , piece_id=GM7TCP65VSFAGWDCRG4VKWAPJHU5HGZFZTVAD4XM6PJFMLCC4NRA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312038483 (594h58m28.183121528s) [noise=true , address=77.90.31.17:4798 , piece_id=HT7E2VVRR2IURW5OIXQ4BLR55QI2BEKJG3VIRG4CABVPCUZRIGWA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312100291 (594h58m28.183183326s) [noise=true , address=77.90.31.19:4859 , piece_id=HEZMENV33Y6A6MMMMDIUECW44662O24WTIG2RWCWZL3IGNKWCH2A, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312494201 (594h58m28.183577246s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312500252 (594h58m28.183583287s) [noise=true , address=145.79.227.49:4359 , piece_id=RGCUJ32JS6NXVVLH7HWUD5IZNV3HZPRRVIHEBJEII2JPEENVIKZA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.312899192 (594h58m28.183982227s) [noise=true , address=145.79.227.18:4037 , piece_id=IQZILX3EVK57WA5QGC4VGRYTYWQX37PLIASYIYXIHWBWKGTSEESA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313265008 (594h58m28.184348053s) [noise=true , address=145.79.227.6:4267 , piece_id=ZE5IVJYY2XEW3QOZBBE7LYICAALYG3DZHU6K7UDYUGBHRTWPQYPA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313640203 (594h58m28.184723248s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.313904957 (594h58m28.184988012s) [noise=true , address=45.91.116.14:4213 , piece_id=BUTDQ3PI56Q7ZHNYHCEMWPHJUI42LXBR42ZGF4VKMDXJ47BSPU3Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314038000 (594h58m28.185121045s) [noise=true , address=77.90.31.12:4603 , piece_id=QQ5V6RXCHOHVL2QTWLLRFPJBAQQ5KW47T2N44EBMIUW747Q4ZOAA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314410920 (594h58m28.185493965s) [noise=true , address=145.79.227.44:4219 , piece_id=NSHMZDOI264JBC5ML4GRTH66FKT76C5QQ534UA4QDFEIG5P6CHDQ, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.314664292 (594h58m28.185747338s) [noise=true , address=145.79.227.40:4062 , piece_id=RTSWJJIEADDCF5Z4JMT4NERSF7EXCWP7QZKDN4KIWEX54UDLNZEA, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:28.315164305 (594h58m28.18624734s) [noise=true , address=77.90.31.10:4540 , piece_id=5LXEPFE5MNCGS6KJK4RP5Q5MV4M3TQMVR5FK4DYTYY4MKTFXTP2Q, seg_pos={0 0}, upload=521557]: Uploading piece... Dec 30 10:27:29.778852769 (594h58m29.649935804s) [upload=521557]: Next returned nil segment Dec 30 10:27:29.778904818 (594h58m29.649987863s) [upload=521557]: Waiting for error group managing segments... Dec 30 10:27:30.177976438 (594h58m30.049059474s) [noise=true , address=145.79.227.18:4037 , piece_id=IQZILX3EVK57WA5QGC4VGRYTYWQX37PLIASYIYXIHWBWKGTSEESA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.239401413 (594h58m30.110484448s) [noise=true , address=145.79.227.6:4267 , piece_id=ZE5IVJYY2XEW3QOZBBE7LYICAALYG3DZHU6K7UDYUGBHRTWPQYPA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.247231673 (594h58m30.118314708s) [noise=true , address=145.79.227.35:4492 , piece_id=GM7TCP65VSFAGWDCRG4VKWAPJHU5HGZFZTVAD4XM6PJFMLCC4NRA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.364460186 (594h58m30.235543221s) [noise=true , address=77.90.31.17:4798 , piece_id=HT7E2VVRR2IURW5OIXQ4BLR55QI2BEKJG3VIRG4CABVPCUZRIGWA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.415614833 (594h58m30.286697868s) [noise=true , address=77.90.31.12:4603 , piece_id=QQ5V6RXCHOHVL2QTWLLRFPJBAQQ5KW47T2N44EBMIUW747Q4ZOAA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.444886882 (594h58m30.315969917s) [noise=true , address=77.90.31.10:4540 , piece_id=5LXEPFE5MNCGS6KJK4RP5Q5MV4M3TQMVR5FK4DYTYY4MKTFXTP2Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.512047427 (594h58m30.383130462s) [noise=true , address=45.91.116.14:4213 , piece_id=BUTDQ3PI56Q7ZHNYHCEMWPHJUI42LXBR42ZGF4VKMDXJ47BSPU3Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.536847771 (594h58m30.407930806s) [noise=true , address=77.90.31.19:4859 , piece_id=HEZMENV33Y6A6MMMMDIUECW44662O24WTIG2RWCWZL3IGNKWCH2A, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:30.680580563 (594h58m30.551663608s) [noise=true , address=45.91.116.12:4253 , piece_id=NPLIO7NLEIRTQAWDQ7WZVZE5ZSRLDPDNM34M6SEI53G5XEHB3LJA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.065501378 (594h58m30.936584414s) [noise=true , address=145.79.227.49:4359 , piece_id=RGCUJ32JS6NXVVLH7HWUD5IZNV3HZPRRVIHEBJEII2JPEENVIKZA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.104555514 (594h58m30.975638559s) [noise=true , address=145.79.227.44:4219 , piece_id=NSHMZDOI264JBC5ML4GRTH66FKT76C5QQ534UA4QDFEIG5P6CHDQ, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.115841635 (594h58m30.98692467s) [noise=true , address=145.79.227.30:4783 , piece_id=UIXZYYXWC7KJZUV3BNZAIUSZNDEYA5RFFLKMXWF3XLNKJF5K2A6Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.364792710 (594h58m31.235875755s) [noise=true , address=145.79.227.40:4062 , piece_id=RTSWJJIEADDCF5Z4JMT4NERSF7EXCWP7QZKDN4KIWEX54UDLNZEA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.374552906 (594h58m31.245635941s) [noise=true , address=145.79.227.43:4179 , piece_id=CQ32O4X3PKQN37OWC3WU5Z5FF7CCAIOAWLUISOXJTTIXTSMKYNQA, seg_pos={0 0}, upload=521557]: Done uploading piece. err: <nil> Dec 30 10:27:31.374572984 (594h58m31.245656019s) [seg_pos={0 0}, upload=521557]: Segment reached optimal threshold of 14 pieces. Dec 30 10:27:31.374775900 (594h58m31.245858935s) [noise=true , address=45.91.116.10:4115 , piece_id=DRZ2YOFTPSE7SQD6MJ6YQWRQN6MDOLHHP7LWJMKMCPHO37A72C4A, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1arcuTW5rwnvfiqi2H7dL3oL6jKBkDeGq6DmgRGpS9Vs9BCHmf): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.374782553 (594h58m31.245865598s) [noise=true , address=45.91.116.10:4115 , piece_id=DRZ2YOFTPSE7SQD6MJ6YQWRQN6MDOLHHP7LWJMKMCPHO37A72C4A, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.374972405 (594h58m31.24605544s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQCT2yf3pZep8VeNBcKma579ot9EfWgaNkXQNofKjHTzdCuoc): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.374978286 (594h58m31.246061321s) [noise=true , address=77.90.31.11:4575 , piece_id=UZEQEFJKQVI4WRGN5T7GVFU7HZNF7B4RBLJZ6CQUCX37D3OTP2ZQ, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375019294 (594h58m31.246102349s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo): protocol: expected piece hash; context canceled; EOF Dec 30 10:27:31.375038921 (594h58m31.246121966s) [noise=true , address=145.79.227.33:4873 , piece_id=EKKHHG4PBR2YGSGDXRRHDEEWF6IMSJB6O24Q4DGWWA5RFIGFZU2Q, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375091411 (594h58m31.246174446s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12W5dFL7EEUwuze62HrcyCpVV7zZUjGnJH6PVHq4cQEEwTKPujE): protocol: write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection; write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52208->77.90.31.21:4896: use of closed network connection Dec 30 10:27:31.375099606 (594h58m31.246182641s) [noise=true , address=77.90.31.21:4896 , piece_id=WN5XOACLQOTWYULRB7GVPSZXTA6BKJN3N6FPICIINB77REE43N6Q, seg_pos={0 0}, upload=521557]: Long tail cancellation due to optimal threshold Dec 30 10:27:31.375195900 (594h58m31.246278935s) [seg_pos={0 0}, upload=521557]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 10:27:31.375261705 (594h58m31.24634474s) [upload=521557]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 10:27:31.375266604 (594h58m31.246349639s) [upload=521557]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 10:27:31.664767884 (594h58m31.53585099s) [upload=521557]: Done uploading segments... Dec 30 10:27:31.664793884 (594h58m31.535876929s) [upload=521557]: Upload finished. err: <nil> Duration: 15.319411465s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)