Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X....X........................................X..] duration: 37.613438468s trace: https://jaeger.dev.storj.io/trace/1ebdd84a47b0fea8 FAIL (2026-01-01 03:41:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3 , piece_id=GP4LIOS53UP3P4JAJSMKZRENL66UKKAYPG5Z5LN6AQJQM7RKFPWQ, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.555732730 (636h10m14.426815775s) [noise=true , address=45.91.116.8:4068 , piece_id=2EJJOJHC2EVMRLNTSFHTCCWHUACTM3UCI7557BLV6AGCIKZAIM3Q, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.555861985 (636h10m14.42694503s) [noise=true , address=145.79.227.34:4934 , piece_id=RPORZ7NEJGQK7WXN5PSBXHLYPIM3LAYN4R6GI6L4TZCRDTPTOWCA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.556167417 (636h10m14.427250462s) [noise=true , address=145.79.227.33:4874 , piece_id=ISLR7HHUN5C4JCKEMKIXFJ7X53SK5JJFHAG7SQ5UCIHZMX7FZV6A, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.557132745 (636h10m14.42821578s) [noise=true , address=145.79.227.4:4215 , piece_id=5ZN6S74EDURKEE4QZ7FEOSVOY3NIGK45JUHBJMNGLDWTQUVYQHHA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.557290145 (636h10m14.42837318s) [noise=true , address=64.49.13.169:4710 , piece_id=XOLOSFJMCEWHMYC46T5BD3RUSDZ77AFPFADVEQCDWGRV2JCKGINQ, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.558546016 (636h10m14.429629051s) [noise=true , address=145.79.227.30:4779 , piece_id=IP2UMBX5PXIDW5AOWSZYLD42TKJSWPHDZQYVCH2AZ4LHGXAAWLXA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.559810844 (636h10m14.430893889s) [noise=true , address=145.79.227.44:4203 , piece_id=UCRXAICCLNAQJGY4DWPEJSYG3Q2YU2UNJSIMTGQ6VQFVW2BJ5A2Q, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.561308566 (636h10m14.432391601s) [noise=true , address=145.79.227.10:4421 , piece_id=3SDLFTSJH5W332QOTMC345XUN25LYPBP6HMKTY5ENOEO2WZEGPUA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.562571591 (636h10m14.433654636s) [noise=true , address=145.79.227.43:4164 , piece_id=4J7RF32DYWIYH7NTAPA7WNNXBXQS5B6JOXSDUTLLCMWDDR6IN7OQ, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.563473288 (636h10m14.434556333s) [noise=true , address=145.79.227.40:4053 , piece_id=F73PCPUM44L7KECQWIOKZIX67AVART4WFDM63UEUDV7F3Q66YOJA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.564781790 (636h10m14.435864825s) [noise=true , address=77.90.31.10:4546 , piece_id=OEVEQHYJBGE2FIABEPUYMSOS2NBAM6ZRPQ4J5PAQR3G6QRPS43VA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.565835766 (636h10m14.436918801s) [noise=true , address=145.79.227.41:4107 , piece_id=POHERAN6H5M3HCP5YGK4IKFPUV3JII5QV5QZA6CQRD77XEFGNFFQ, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.567015342 (636h10m14.438098387s) [noise=true , address=145.79.227.2:4118 , piece_id=W3EA45ZCGH5IAZAZIXZ6GVW6SYAOPN3WDX4DLMAESZIRXXVZ65QA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.567628530 (636h10m14.438711565s) [noise=true , address=77.90.31.13:4654 , piece_id=ND5P6ZAMBBNT6OFNRIAUORAVPMC45CB6COFGZNZV4WCJKNDKSZBA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:16.014065836 (636h10m15.885148871s) [upload=557441]: Next returned nil segment Jan 1 03:39:16.014083580 (636h10m15.885166615s) [upload=557441]: Waiting for error group managing segments... Jan 1 03:39:16.455400629 (636h10m16.326483664s) [noise=true , address=145.79.227.10:4421 , piece_id=3SDLFTSJH5W332QOTMC345XUN25LYPBP6HMKTY5ENOEO2WZEGPUA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:16.464872484 (636h10m16.335955529s) [noise=true , address=145.79.227.4:4215 , piece_id=5ZN6S74EDURKEE4QZ7FEOSVOY3NIGK45JUHBJMNGLDWTQUVYQHHA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:16.616875776 (636h10m16.487958821s) [noise=true , address=77.90.31.19:4876 , piece_id=UVRE5SNVM7SVUWD37QXCFDL7FWRQIMGFBYKLUULJKQJ4SW6224PA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:16.759240917 (636h10m16.630323952s) [noise=true , address=77.90.31.21:4907 , piece_id=XELNGQJ3EHMREVTXIFRR4FLXDD5U6CMAQ53GRSJ4WOT76QOQCBGQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:16.826797803 (636h10m16.697880848s) [noise=true , address=77.90.31.10:4546 , piece_id=OEVEQHYJBGE2FIABEPUYMSOS2NBAM6ZRPQ4J5PAQR3G6QRPS43VA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.039826124 (636h10m16.910909169s) [noise=true , address=145.79.227.49:4357 , piece_id=NRGFOQVKNCENLBXQDAU63ZQTJXYDJZ5YMEMZUHNBAWCERZCDI5EQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.319810887 (636h10m17.190893922s) [noise=true , address=145.79.227.30:4779 , piece_id=IP2UMBX5PXIDW5AOWSZYLD42TKJSWPHDZQYVCH2AZ4LHGXAAWLXA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.335275864 (636h10m17.206358899s) [noise=true , address=145.79.227.34:4934 , piece_id=RPORZ7NEJGQK7WXN5PSBXHLYPIM3LAYN4R6GI6L4TZCRDTPTOWCA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.360870549 (636h10m17.231953594s) [noise=true , address=145.79.227.40:4053 , piece_id=F73PCPUM44L7KECQWIOKZIX67AVART4WFDM63UEUDV7F3Q66YOJA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.412439583 (636h10m17.283522628s) [noise=true , address=145.79.227.35:4513 , piece_id=GP4LIOS53UP3P4JAJSMKZRENL66UKKAYPG5Z5LN6AQJQM7RKFPWQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.469220361 (636h10m17.340303396s) [noise=true , address=145.79.227.44:4203 , piece_id=UCRXAICCLNAQJGY4DWPEJSYG3Q2YU2UNJSIMTGQ6VQFVW2BJ5A2Q, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.485589420 (636h10m17.356672455s) [noise=true , address=145.79.227.33:4874 , piece_id=ISLR7HHUN5C4JCKEMKIXFJ7X53SK5JJFHAG7SQ5UCIHZMX7FZV6A, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.598713200 (636h10m17.469796245s) [noise=true , address=145.79.227.43:4164 , piece_id=4J7RF32DYWIYH7NTAPA7WNNXBXQS5B6JOXSDUTLLCMWDDR6IN7OQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.911962313 (636h10m17.783045348s) [noise=true , address=145.79.227.41:4107 , piece_id=POHERAN6H5M3HCP5YGK4IKFPUV3JII5QV5QZA6CQRD77XEFGNFFQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.911978394 (636h10m17.783061429s) [seg_pos={0 0}, upload=557441]: Segment reached optimal threshold of 14 pieces. Jan 1 03:39:17.912112600 (636h10m17.783195635s) [noise=true , address=77.90.31.13:4654 , piece_id=ND5P6ZAMBBNT6OFNRIAUORAVPMC45CB6COFGZNZV4WCJKNDKSZBA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: expected piece hash; context canceled; EOF Jan 1 03:39:17.912118220 (636h10m17.783201255s) [noise=true , address=77.90.31.13:4654 , piece_id=ND5P6ZAMBBNT6OFNRIAUORAVPMC45CB6COFGZNZV4WCJKNDKSZBA, seg_pos={0 0}, upload=557441]: Long tail cancellation due to optimal threshold Jan 1 03:39:17.912348508 (636h10m17.783431543s) [noise=true , address=45.91.116.8:4068 , piece_id=2EJJOJHC2EVMRLNTSFHTCCWHUACTM3UCI7557BLV6AGCIKZAIM3Q, seg_pos={0 0}, upload=557441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Jan 1 03:39:17.912354901 (636h10m17.783437936s) [noise=true , address=45.91.116.8:4068 , piece_id=2EJJOJHC2EVMRLNTSFHTCCWHUACTM3UCI7557BLV6AGCIKZAIM3Q, seg_pos={0 0}, upload=557441]: Long tail cancellation due to optimal threshold Jan 1 03:39:17.912458238 (636h10m17.783541283s) [noise=true , address=64.49.13.169:4710 , piece_id=XOLOSFJMCEWHMYC46T5BD3RUSDZ77AFPFADVEQCDWGRV2JCKGINQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:55400->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:55400->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55400->64.49.13.169:4710: use of closed network connection Jan 1 03:39:17.912463217 (636h10m17.783546252s) [noise=true , address=64.49.13.169:4710 , piece_id=XOLOSFJMCEWHMYC46T5BD3RUSDZ77AFPFADVEQCDWGRV2JCKGINQ, seg_pos={0 0}, upload=557441]: Long tail cancellation due to optimal threshold Jan 1 03:39:17.912551856 (636h10m17.783634891s) [noise=true , address=145.79.227.2:4118 , piece_id=W3EA45ZCGH5IAZAZIXZ6GVW6SYAOPN3WDX4DLMAESZIRXXVZ65QA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ty5LHXwLmHC7nRPmVfj2wVEif6JQAQQ5Z5218vv1fKK5sD3zE): protocol: write tcp 188.166.225.170:54838->145.79.227.2:4118: use of closed network connection; write tcp 188.166.225.170:54838->145.79.227.2:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54838->145.79.227.2:4118: use of closed network connection Jan 1 03:39:17.912557115 (636h10m17.783640151s) [noise=true , address=145.79.227.2:4118 , piece_id=W3EA45ZCGH5IAZAZIXZ6GVW6SYAOPN3WDX4DLMAESZIRXXVZ65QA, seg_pos={0 0}, upload=557441]: Long tail cancellation due to optimal threshold Jan 1 03:39:17.912582464 (636h10m17.783665499s) [seg_pos={0 0}, upload=557441]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 03:39:17.912645484 (636h10m17.783728519s) [upload=557441]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:39:17.912649532 (636h10m17.783732567s) [upload=557441]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:39:18.208729342 (636h10m18.079812378s) [upload=557441]: Done uploading segments... Jan 1 03:39:18.208744913 (636h10m18.079827958s) [upload=557441]: Upload finished. err: <nil> Duration: 37.613462443s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [X...X..X..X...X..X....XXX.X.X..XX.X.XXX..XX..XXXXX....XX...X] duration: 30.003565291s trace: https://jaeger.dev.storj.io/trace/578e2d38a7451922 FAIL (2026-01-01 03:41:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 1 03:41:23.760805468 (636h12m23.631888503s) [upload=557472]: Starting upload Jan 1 03:41:23.760931168 (636h12m23.632014203s) [upload=557472]: Done starting upload Jan 1 03:41:23.761059753 (636h12m23.632142788s) [upload=557472]: Uploading segments... Jan 1 03:41:23.761244134 (636h12m23.632327179s) [upload=557472]: Got next segment. Inline: false Jan 1 03:41:23.761289019 (636h12m23.632372064s) [upload=557472]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:41:23.761294679 (636h12m23.632377725s) [upload=557472]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:41:24.005225724 (636h12m23.876308759s) [seg_pos={0 0}, upload=557472]: Begin upload segment... Jan 1 03:41:24.005265580 (636h12m23.876348625s) [seg_pos={0 0}, upload=557472]: Done begin upload segment. Jan 1 03:41:24.005302119 (636h12m23.876385154s) [noise=true , address=145.79.227.44:4193 , piece_id=37VO3UL7VLUUPDQX77IZSQRFIKUD5PWXCCLTIDZHW46NQ3DHNBQA, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.005319352 (636h12m23.876402387s) [noise=true , address=145.79.227.6:4258 , piece_id=SMO2K34YK4DVDZGLROCBJX6JYMZNEEFSISWIMOWR6CTN5A3467EQ, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.005947307 (636h12m23.877030352s) [noise=true , address=145.79.227.12:4765 , piece_id=CJBYBAU2IOR2PMBSVZE5OPYJKMBKJVZTSU7FIFIR736PGS5KWZWQ, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.005958199 (636h12m23.877041244s) [noise=true , address=145.79.227.38:4977 , piece_id=26ZRSSF4XZEESUMZB5TE4PSRM3BWT27LZJN7LEKDVMWRQMGYF43Q, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.006293477 (636h12m23.877376512s) [noise=true , address=145.79.227.8:4330 , piece_id=VFJVHZGG5IADJ3QOPENHZ3BS3E7225IEBK3J3JDY6GK7FBJDJWKA, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.006383919 (636h12m23.877466964s) [noise=true , address=145.79.227.37:4566 , piece_id=7VAGGFQZC76L7OW2RKJ6ICX3SHGTDEJT42XPYXUGWOQM6X6E2SEQ, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.006580674 (636h12m23.877663719s) [noise=true , address=145.79.227.11:4730 , piece_id=MSTE7YQULV46R7GZGHXHOP7BZECL5RHVILPDPJXG6CWRYDBWBUHQ, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.006798148 (636h12m23.877881193s) [noise=true , address=45.91.116.7:4010 , piece_id=55VM6V4D3TO7FMWJDPDERFFWGHYOEEEDCM3DU5SER46RHC3EKR6Q, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.006928547 (636h12m23.878011592s) [noise=true , address=145.79.227.3:4151 , piece_id=WUVJJQEZXID5SOCQ2H264P5I2K2LWK6QFTAOU6SZZJ7YHI2QQBPA, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.007226243 (636h12m23.878309288s) [noise=true , address=145.79.227.51:4410 , piece_id=SLNW7BQVL7O3NQJXZ3LR737T76WIZKEIIGIWW5EY7BIASW5OTLMA, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.007241121 (636h12m23.878324186s) [noise=true , address=77.90.31.17:4773 , piece_id=RQAKIN7NOO2OBS7Z4KAJPWPOY6INH7RAHMESSLMS5PVC4YX7EK6A, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.007629291 (636h12m23.878712336s) [noise=true , address=77.90.31.12:4590 , piece_id=M4VFD4RD467ZP7DXPOM76DL37ET6QTXGLYAMIMJPMFBZEGCRQG6Q, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.007771181 (636h12m23.878854226s) [noise=true , address=145.79.227.27:4689 , piece_id=ECCFQVZDR2VD2XYLQPPSCSTFTJP3IPHFKCIV2FWPOHIUXU7DATSQ, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.008011939 (636h12m23.879094974s) [noise=true , address=45.91.116.6:4976 , piece_id=Y2CH7CJ6S7B2TJWBMRFE5R3BQXTLN5ULEKTI7CVFDARQXU72L5NQ, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.008198785 (636h12m23.87928183s) [noise=true , address=77.90.31.10:4517 , piece_id=A5KUMBH7BWYGXF744BTA4ZEH4A6ZIUKBOJSVGKVOQM6HPRT2JU6A, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:24.009086705 (636h12m23.88016975s) [noise=true , address=45.91.116.14:4189 , piece_id=OO5UWBNKAM2JR2HRCBKCOARVBGAQGJYDF375KFZ4AWHSB6E3QOKQ, seg_pos={0 0}, upload=557472]: Uploading piece... Jan 1 03:41:25.709979115 (636h12m25.58106216s) [upload=557472]: Next returned nil segment Jan 1 03:41:25.709996629 (636h12m25.581079664s) [upload=557472]: Waiting for error group managing segments... Jan 1 03:41:25.996252633 (636h12m25.867335678s) [noise=true , address=145.79.227.38:4977 , piece_id=26ZRSSF4XZEESUMZB5TE4PSRM3BWT27LZJN7LEKDVMWRQMGYF43Q, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:25.996528799 (636h12m25.867611844s) [noise=true , address=145.79.227.6:4258 , piece_id=SMO2K34YK4DVDZGLROCBJX6JYMZNEEFSISWIMOWR6CTN5A3467EQ, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:25.998327114 (636h12m25.869410159s) [noise=true , address=145.79.227.3:4151 , piece_id=WUVJJQEZXID5SOCQ2H264P5I2K2LWK6QFTAOU6SZZJ7YHI2QQBPA, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:26.001891932 (636h12m25.872974967s) [noise=true , address=145.79.227.37:4566 , piece_id=7VAGGFQZC76L7OW2RKJ6ICX3SHGTDEJT42XPYXUGWOQM6X6E2SEQ, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:26.003898122 (636h12m25.874981167s) [noise=true , address=145.79.227.51:4410 , piece_id=SLNW7BQVL7O3NQJXZ3LR737T76WIZKEIIGIWW5EY7BIASW5OTLMA, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:26.010471007 (636h12m25.881554052s) [noise=true , address=145.79.227.12:4765 , piece_id=CJBYBAU2IOR2PMBSVZE5OPYJKMBKJVZTSU7FIFIR736PGS5KWZWQ, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:26.029444585 (636h12m25.90052762s) [noise=true , address=145.79.227.11:4730 , piece_id=MSTE7YQULV46R7GZGHXHOP7BZECL5RHVILPDPJXG6CWRYDBWBUHQ, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:26.035855903 (636h12m25.906938938s) [noise=true , address=145.79.227.8:4330 , piece_id=VFJVHZGG5IADJ3QOPENHZ3BS3E7225IEBK3J3JDY6GK7FBJDJWKA, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:26.172690202 (636h12m26.043773247s) [noise=true , address=145.79.227.27:4689 , piece_id=ECCFQVZDR2VD2XYLQPPSCSTFTJP3IPHFKCIV2FWPOHIUXU7DATSQ, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:26.299178520 (636h12m26.170261565s) [noise=true , address=45.91.116.6:4976 , piece_id=Y2CH7CJ6S7B2TJWBMRFE5R3BQXTLN5ULEKTI7CVFDARQXU72L5NQ, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:26.348111114 (636h12m26.219194149s) [noise=true , address=45.91.116.7:4010 , piece_id=55VM6V4D3TO7FMWJDPDERFFWGHYOEEEDCM3DU5SER46RHC3EKR6Q, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:26.428664436 (636h12m26.299747471s) [noise=true , address=45.91.116.14:4189 , piece_id=OO5UWBNKAM2JR2HRCBKCOARVBGAQGJYDF375KFZ4AWHSB6E3QOKQ, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:26.871462616 (636h12m26.742545651s) [noise=true , address=77.90.31.10:4517 , piece_id=A5KUMBH7BWYGXF744BTA4ZEH4A6ZIUKBOJSVGKVOQM6HPRT2JU6A, seg_pos={0 0}, upload=557472]: Done uploading piece. err: <nil> Jan 1 03:41:26.871483817 (636h12m26.742566852s) [seg_pos={0 0}, upload=557472]: Segment reached optimal threshold of 13 pieces. Jan 1 03:41:26.871654852 (636h12m26.742737887s) [noise=true , address=145.79.227.44:4193 , piece_id=37VO3UL7VLUUPDQX77IZSQRFIKUD5PWXCCLTIDZHW46NQ3DHNBQA, seg_pos={0 0}, upload=557472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Jan 1 03:41:26.871661876 (636h12m26.742744911s) [noise=true , address=145.79.227.44:4193 , piece_id=37VO3UL7VLUUPDQX77IZSQRFIKUD5PWXCCLTIDZHW46NQ3DHNBQA, seg_pos={0 0}, upload=557472]: Long tail cancellation due to optimal threshold Jan 1 03:41:26.871854993 (636h12m26.742938028s) [noise=true , address=77.90.31.12:4590 , piece_id=M4VFD4RD467ZP7DXPOM76DL37ET6QTXGLYAMIMJPMFBZEGCRQG6Q, seg_pos={0 0}, upload=557472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Jan 1 03:41:26.871883688 (636h12m26.742966733s) [noise=true , address=77.90.31.12:4590 , piece_id=M4VFD4RD467ZP7DXPOM76DL37ET6QTXGLYAMIMJPMFBZEGCRQG6Q, seg_pos={0 0}, upload=557472]: Long tail cancellation due to optimal threshold Jan 1 03:41:26.871895791 (636h12m26.742978826s) [noise=true , address=77.90.31.17:4773 , piece_id=RQAKIN7NOO2OBS7Z4KAJPWPOY6INH7RAHMESSLMS5PVC4YX7EK6A, seg_pos={0 0}, upload=557472]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Jan 1 03:41:26.871904658 (636h12m26.742987694s) [noise=true , address=77.90.31.17:4773 , piece_id=RQAKIN7NOO2OBS7Z4KAJPWPOY6INH7RAHMESSLMS5PVC4YX7EK6A, seg_pos={0 0}, upload=557472]: Long tail cancellation due to optimal threshold Jan 1 03:41:26.871947390 (636h12m26.743030425s) [seg_pos={0 0}, upload=557472]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:41:26.872014868 (636h12m26.743097903s) [upload=557472]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:41:26.872020168 (636h12m26.743103203s) [upload=557472]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:41:27.158644383 (636h12m27.029727418s) [upload=557472]: Done uploading segments... Jan 1 03:41:27.158663450 (636h12m27.029746485s) [upload=557472]: Upload finished. err: <nil> Duration: 30.003582795s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [............X......X............................X........X..] duration: 3.569678078s trace: https://jaeger.dev.storj.io/trace/8b9520ee0ddc710 FAIL (2026-01-01 03:41:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.569766146s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.X...X..X..X...X..X....XXX.X.X..XX.X.XXX..XX..XXXXX....XX...] duration: 6.012484162s trace: https://jaeger.dev.storj.io/trace/3a1918a7502a0c40 FAIL (2026-01-01 03:41:07 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 1 03:38:18.274492318 (636h9m18.145575353s) [upload=557425]: Starting upload Jan 1 03:38:18.274567070 (636h9m18.145650105s) [upload=557425]: Done starting upload Jan 1 03:38:18.274688662 (636h9m18.145771707s) [upload=557425]: Uploading segments... Jan 1 03:38:18.274806416 (636h9m18.145889461s) [upload=557425]: Got next segment. Inline: false Jan 1 03:38:18.274842244 (636h9m18.145925289s) [upload=557425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:38:18.274847464 (636h9m18.145930519s) [upload=557425]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:38:18.522138205 (636h9m18.39322127s) [seg_pos={0 0}, upload=557425]: Begin upload segment... Jan 1 03:38:18.522195464 (636h9m18.393278509s) [seg_pos={0 0}, upload=557425]: Done begin upload segment. Jan 1 03:38:18.522222756 (636h9m18.393305791s) [noise=true , address=145.79.227.15:4874 , piece_id=LCSGCVZS3FMNO3EG45DQRE4WIB6356TDEYCAM7SFEQIRGNMMZHVQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.522240590 (636h9m18.393323655s) [noise=true , address=145.79.227.5:4221 , piece_id=LR5HLZ2FXOW2VBFYUH7N4NCEX46VWHJIKXXT7VQCOXH3R2PY5H6Q, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.522841954 (636h9m18.393924999s) [noise=true , address=45.91.116.6:4976 , piece_id=5RDOOLXQNMLR2Z5XSV4YI42ZBKPPPHLIFDZ6WBUG6T5CKGSFH3BQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523203123 (636h9m18.394286168s) [noise=true , address=45.91.116.8:4046 , piece_id=AATWWQ44SIMWQETNR37O7OSCOQFOIRSQDW7QTVJOTM75FBD2TGXQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523238200 (636h9m18.394321245s) [noise=true , address=145.79.227.19:4077 , piece_id=4R4DDZL5TEPSTNNLYMWZYPWJX62WM3JKV4UHBQYQQIJMEOS4KR6A, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523581053 (636h9m18.394664088s) [noise=true , address=145.79.227.37:4567 , piece_id=6UKCBEODRCAEAWW2UXHFQHVUBLBTFEGFPJXZH27EFBKALQCS34NA, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523675452 (636h9m18.394758487s) [noise=true , address=145.79.227.47:4266 , piece_id=BYNWCGK4PXBHSE4SNQUT24MUOPBXUFIY225736NRWKNLXEKJ5LDA, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523829435 (636h9m18.39491247s) [noise=true , address=145.79.227.17:4970 , piece_id=4QCPG54UE4X3CXT6H2UOQPEO4WHGMVZM37V6YFN5RDCKRGAYAGDQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523980252 (636h9m18.395063287s) [noise=true , address=145.79.227.22:4509 , piece_id=I4F3MQKLEZRE2YC37V2TQQO7E2XNOPTWKLHEDLYXGPJE2NVQCNEQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524079642 (636h9m18.395162677s) [noise=true , address=145.79.227.18:4007 , piece_id=2A2MEUNPIIQR4FBUEVGOY4HO5MGV6LAOFODMEDEO6OQRUVLK62AA, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524396465 (636h9m18.3954795s) [noise=true , address=145.79.227.29:4942 , piece_id=ZJEEMQFL7RKIOB3OILJD47A6LHRDVTLK66X34KVFQ46H7RU7MPSQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524481868 (636h9m18.395564943s) [noise=true , address=77.90.31.12:4589 , piece_id=FGSJYHVF6QRKWSF6DOP7DMXMRF64BBMIBVX7VPPMFXJOFSQ3C47A, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524653404 (636h9m18.395736439s) [noise=true , address=45.91.116.10:4083 , piece_id=QITJ4BWAMDX5G2M5R5HTTS5A3ZHWCJQI56HM66CRSW4IP4PAJKNQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524906376 (636h9m18.395989411s) [noise=true , address=145.79.227.36:4519 , piece_id=IEXN5NPPGDTELLYLLZYMCUYENTIN6A4N4ALEPJIEHSF4INKNTPAQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524912417 (636h9m18.395995452s) [noise=true , address=145.79.227.11:4731 , piece_id=MXYDHPFONCMBGPUXEO2DRLOAZQ45VRFLCZP6MXBQFEUAHVS2HEQQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.525774960 (636h9m18.396857995s) [noise=true , address=145.79.227.46:4601 , piece_id=R2ZUOVU3KQIVFEYWJLTX6BBZVPEFKE5LMSDTWGGOZDTEW2NORMKQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:20.226070182 (636h9m20.097153217s) [upload=557425]: Next returned nil segment Jan 1 03:38:20.226084649 (636h9m20.097167684s) [upload=557425]: Waiting for error group managing segments... Jan 1 03:38:20.500808796 (636h9m20.371891841s) [noise=true , address=145.79.227.11:4731 , piece_id=MXYDHPFONCMBGPUXEO2DRLOAZQ45VRFLCZP6MXBQFEUAHVS2HEQQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.511146620 (636h9m20.382229665s) [noise=true , address=145.79.227.46:4601 , piece_id=R2ZUOVU3KQIVFEYWJLTX6BBZVPEFKE5LMSDTWGGOZDTEW2NORMKQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.512963139 (636h9m20.384046184s) [noise=true , address=145.79.227.5:4221 , piece_id=LR5HLZ2FXOW2VBFYUH7N4NCEX46VWHJIKXXT7VQCOXH3R2PY5H6Q, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.513585224 (636h9m20.384668269s) [noise=true , address=145.79.227.15:4874 , piece_id=LCSGCVZS3FMNO3EG45DQRE4WIB6356TDEYCAM7SFEQIRGNMMZHVQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.517155291 (636h9m20.388238336s) [noise=true , address=145.79.227.17:4970 , piece_id=4QCPG54UE4X3CXT6H2UOQPEO4WHGMVZM37V6YFN5RDCKRGAYAGDQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.528961282 (636h9m20.400044317s) [noise=true , address=145.79.227.19:4077 , piece_id=4R4DDZL5TEPSTNNLYMWZYPWJX62WM3JKV4UHBQYQQIJMEOS4KR6A, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.530863914 (636h9m20.401946949s) [noise=true , address=145.79.227.22:4509 , piece_id=I4F3MQKLEZRE2YC37V2TQQO7E2XNOPTWKLHEDLYXGPJE2NVQCNEQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.538983445 (636h9m20.41006649s) [noise=true , address=145.79.227.18:4007 , piece_id=2A2MEUNPIIQR4FBUEVGOY4HO5MGV6LAOFODMEDEO6OQRUVLK62AA, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.595792125 (636h9m20.46687516s) [noise=true , address=145.79.227.36:4519 , piece_id=IEXN5NPPGDTELLYLLZYMCUYENTIN6A4N4ALEPJIEHSF4INKNTPAQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.612204215 (636h9m20.48328725s) [noise=true , address=145.79.227.47:4266 , piece_id=BYNWCGK4PXBHSE4SNQUT24MUOPBXUFIY225736NRWKNLXEKJ5LDA, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.767931659 (636h9m20.639014694s) [noise=true , address=77.90.31.12:4589 , piece_id=FGSJYHVF6QRKWSF6DOP7DMXMRF64BBMIBVX7VPPMFXJOFSQ3C47A, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.867883629 (636h9m20.738966664s) [noise=true , address=45.91.116.10:4083 , piece_id=QITJ4BWAMDX5G2M5R5HTTS5A3ZHWCJQI56HM66CRSW4IP4PAJKNQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.871885079 (636h9m20.742968114s) [noise=true , address=45.91.116.8:4046 , piece_id=AATWWQ44SIMWQETNR37O7OSCOQFOIRSQDW7QTVJOTM75FBD2TGXQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.871902772 (636h9m20.742985807s) [seg_pos={0 0}, upload=557425]: Segment reached optimal threshold of 13 pieces. Jan 1 03:38:20.872093115 (636h9m20.74317618s) [noise=true , address=145.79.227.29:4942 , piece_id=ZJEEMQFL7RKIOB3OILJD47A6LHRDVTLK66X34KVFQ46H7RU7MPSQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Jan 1 03:38:20.872105989 (636h9m20.743189024s) [noise=true , address=145.79.227.29:4942 , piece_id=ZJEEMQFL7RKIOB3OILJD47A6LHRDVTLK66X34KVFQ46H7RU7MPSQ, seg_pos={0 0}, upload=557425]: Long tail cancellation due to optimal threshold Jan 1 03:38:20.872304367 (636h9m20.743387402s) [noise=true , address=45.91.116.6:4976 , piece_id=5RDOOLXQNMLR2Z5XSV4YI42ZBKPPPHLIFDZ6WBUG6T5CKGSFH3BQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 1 03:38:20.872325717 (636h9m20.743408772s) [noise=true , address=45.91.116.6:4976 , piece_id=5RDOOLXQNMLR2Z5XSV4YI42ZBKPPPHLIFDZ6WBUG6T5CKGSFH3BQ, seg_pos={0 0}, upload=557425]: Long tail cancellation due to optimal threshold Jan 1 03:38:20.872342800 (636h9m20.743425946s) [noise=true , address=145.79.227.37:4567 , piece_id=6UKCBEODRCAEAWW2UXHFQHVUBLBTFEGFPJXZH27EFBKALQCS34NA, seg_pos={0 0}, upload=557425]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:50636->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:50636->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50636->145.79.227.37:4567: use of closed network connection Jan 1 03:38:20.872355464 (636h9m20.743438509s) [noise=true , address=145.79.227.37:4567 , piece_id=6UKCBEODRCAEAWW2UXHFQHVUBLBTFEGFPJXZH27EFBKALQCS34NA, seg_pos={0 0}, upload=557425]: Long tail cancellation due to optimal threshold Jan 1 03:38:20.872439024 (636h9m20.743522069s) [seg_pos={0 0}, upload=557425]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:38:20.872567558 (636h9m20.743650603s) [upload=557425]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:38:20.872573520 (636h9m20.743656555s) [upload=557425]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:38:21.172543628 (636h9m21.043626673s) [upload=557425]: Done uploading segments... Jan 1 03:38:21.172563316 (636h9m21.043646361s) [upload=557425]: Upload finished. err: <nil> Duration: 6.012516174s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X....X........................................X.] duration: 35.947413164s trace: https://jaeger.dev.storj.io/trace/1ebdd84a47b0fea8 FAIL (2026-01-01 03:40:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3 , piece_id=GP4LIOS53UP3P4JAJSMKZRENL66UKKAYPG5Z5LN6AQJQM7RKFPWQ, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.555732730 (636h10m14.426815775s) [noise=true , address=45.91.116.8:4068 , piece_id=2EJJOJHC2EVMRLNTSFHTCCWHUACTM3UCI7557BLV6AGCIKZAIM3Q, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.555861985 (636h10m14.42694503s) [noise=true , address=145.79.227.34:4934 , piece_id=RPORZ7NEJGQK7WXN5PSBXHLYPIM3LAYN4R6GI6L4TZCRDTPTOWCA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.556167417 (636h10m14.427250462s) [noise=true , address=145.79.227.33:4874 , piece_id=ISLR7HHUN5C4JCKEMKIXFJ7X53SK5JJFHAG7SQ5UCIHZMX7FZV6A, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.557132745 (636h10m14.42821578s) [noise=true , address=145.79.227.4:4215 , piece_id=5ZN6S74EDURKEE4QZ7FEOSVOY3NIGK45JUHBJMNGLDWTQUVYQHHA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.557290145 (636h10m14.42837318s) [noise=true , address=64.49.13.169:4710 , piece_id=XOLOSFJMCEWHMYC46T5BD3RUSDZ77AFPFADVEQCDWGRV2JCKGINQ, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.558546016 (636h10m14.429629051s) [noise=true , address=145.79.227.30:4779 , piece_id=IP2UMBX5PXIDW5AOWSZYLD42TKJSWPHDZQYVCH2AZ4LHGXAAWLXA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.559810844 (636h10m14.430893889s) [noise=true , address=145.79.227.44:4203 , piece_id=UCRXAICCLNAQJGY4DWPEJSYG3Q2YU2UNJSIMTGQ6VQFVW2BJ5A2Q, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.561308566 (636h10m14.432391601s) [noise=true , address=145.79.227.10:4421 , piece_id=3SDLFTSJH5W332QOTMC345XUN25LYPBP6HMKTY5ENOEO2WZEGPUA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.562571591 (636h10m14.433654636s) [noise=true , address=145.79.227.43:4164 , piece_id=4J7RF32DYWIYH7NTAPA7WNNXBXQS5B6JOXSDUTLLCMWDDR6IN7OQ, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.563473288 (636h10m14.434556333s) [noise=true , address=145.79.227.40:4053 , piece_id=F73PCPUM44L7KECQWIOKZIX67AVART4WFDM63UEUDV7F3Q66YOJA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.564781790 (636h10m14.435864825s) [noise=true , address=77.90.31.10:4546 , piece_id=OEVEQHYJBGE2FIABEPUYMSOS2NBAM6ZRPQ4J5PAQR3G6QRPS43VA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.565835766 (636h10m14.436918801s) [noise=true , address=145.79.227.41:4107 , piece_id=POHERAN6H5M3HCP5YGK4IKFPUV3JII5QV5QZA6CQRD77XEFGNFFQ, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.567015342 (636h10m14.438098387s) [noise=true , address=145.79.227.2:4118 , piece_id=W3EA45ZCGH5IAZAZIXZ6GVW6SYAOPN3WDX4DLMAESZIRXXVZ65QA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.567628530 (636h10m14.438711565s) [noise=true , address=77.90.31.13:4654 , piece_id=ND5P6ZAMBBNT6OFNRIAUORAVPMC45CB6COFGZNZV4WCJKNDKSZBA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:16.014065836 (636h10m15.885148871s) [upload=557441]: Next returned nil segment Jan 1 03:39:16.014083580 (636h10m15.885166615s) [upload=557441]: Waiting for error group managing segments... Jan 1 03:39:16.455400629 (636h10m16.326483664s) [noise=true , address=145.79.227.10:4421 , piece_id=3SDLFTSJH5W332QOTMC345XUN25LYPBP6HMKTY5ENOEO2WZEGPUA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:16.464872484 (636h10m16.335955529s) [noise=true , address=145.79.227.4:4215 , piece_id=5ZN6S74EDURKEE4QZ7FEOSVOY3NIGK45JUHBJMNGLDWTQUVYQHHA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:16.616875776 (636h10m16.487958821s) [noise=true , address=77.90.31.19:4876 , piece_id=UVRE5SNVM7SVUWD37QXCFDL7FWRQIMGFBYKLUULJKQJ4SW6224PA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:16.759240917 (636h10m16.630323952s) [noise=true , address=77.90.31.21:4907 , piece_id=XELNGQJ3EHMREVTXIFRR4FLXDD5U6CMAQ53GRSJ4WOT76QOQCBGQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:16.826797803 (636h10m16.697880848s) [noise=true , address=77.90.31.10:4546 , piece_id=OEVEQHYJBGE2FIABEPUYMSOS2NBAM6ZRPQ4J5PAQR3G6QRPS43VA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.039826124 (636h10m16.910909169s) [noise=true , address=145.79.227.49:4357 , piece_id=NRGFOQVKNCENLBXQDAU63ZQTJXYDJZ5YMEMZUHNBAWCERZCDI5EQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.319810887 (636h10m17.190893922s) [noise=true , address=145.79.227.30:4779 , piece_id=IP2UMBX5PXIDW5AOWSZYLD42TKJSWPHDZQYVCH2AZ4LHGXAAWLXA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.335275864 (636h10m17.206358899s) [noise=true , address=145.79.227.34:4934 , piece_id=RPORZ7NEJGQK7WXN5PSBXHLYPIM3LAYN4R6GI6L4TZCRDTPTOWCA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.360870549 (636h10m17.231953594s) [noise=true , address=145.79.227.40:4053 , piece_id=F73PCPUM44L7KECQWIOKZIX67AVART4WFDM63UEUDV7F3Q66YOJA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.412439583 (636h10m17.283522628s) [noise=true , address=145.79.227.35:4513 , piece_id=GP4LIOS53UP3P4JAJSMKZRENL66UKKAYPG5Z5LN6AQJQM7RKFPWQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.469220361 (636h10m17.340303396s) [noise=true , address=145.79.227.44:4203 , piece_id=UCRXAICCLNAQJGY4DWPEJSYG3Q2YU2UNJSIMTGQ6VQFVW2BJ5A2Q, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.485589420 (636h10m17.356672455s) [noise=true , address=145.79.227.33:4874 , piece_id=ISLR7HHUN5C4JCKEMKIXFJ7X53SK5JJFHAG7SQ5UCIHZMX7FZV6A, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.598713200 (636h10m17.469796245s) [noise=true , address=145.79.227.43:4164 , piece_id=4J7RF32DYWIYH7NTAPA7WNNXBXQS5B6JOXSDUTLLCMWDDR6IN7OQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.911962313 (636h10m17.783045348s) [noise=true , address=145.79.227.41:4107 , piece_id=POHERAN6H5M3HCP5YGK4IKFPUV3JII5QV5QZA6CQRD77XEFGNFFQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.911978394 (636h10m17.783061429s) [seg_pos={0 0}, upload=557441]: Segment reached optimal threshold of 14 pieces. Jan 1 03:39:17.912112600 (636h10m17.783195635s) [noise=true , address=77.90.31.13:4654 , piece_id=ND5P6ZAMBBNT6OFNRIAUORAVPMC45CB6COFGZNZV4WCJKNDKSZBA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: expected piece hash; context canceled; EOF Jan 1 03:39:17.912118220 (636h10m17.783201255s) [noise=true , address=77.90.31.13:4654 , piece_id=ND5P6ZAMBBNT6OFNRIAUORAVPMC45CB6COFGZNZV4WCJKNDKSZBA, seg_pos={0 0}, upload=557441]: Long tail cancellation due to optimal threshold Jan 1 03:39:17.912348508 (636h10m17.783431543s) [noise=true , address=45.91.116.8:4068 , piece_id=2EJJOJHC2EVMRLNTSFHTCCWHUACTM3UCI7557BLV6AGCIKZAIM3Q, seg_pos={0 0}, upload=557441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Jan 1 03:39:17.912354901 (636h10m17.783437936s) [noise=true , address=45.91.116.8:4068 , piece_id=2EJJOJHC2EVMRLNTSFHTCCWHUACTM3UCI7557BLV6AGCIKZAIM3Q, seg_pos={0 0}, upload=557441]: Long tail cancellation due to optimal threshold Jan 1 03:39:17.912458238 (636h10m17.783541283s) [noise=true , address=64.49.13.169:4710 , piece_id=XOLOSFJMCEWHMYC46T5BD3RUSDZ77AFPFADVEQCDWGRV2JCKGINQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:55400->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:55400->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55400->64.49.13.169:4710: use of closed network connection Jan 1 03:39:17.912463217 (636h10m17.783546252s) [noise=true , address=64.49.13.169:4710 , piece_id=XOLOSFJMCEWHMYC46T5BD3RUSDZ77AFPFADVEQCDWGRV2JCKGINQ, seg_pos={0 0}, upload=557441]: Long tail cancellation due to optimal threshold Jan 1 03:39:17.912551856 (636h10m17.783634891s) [noise=true , address=145.79.227.2:4118 , piece_id=W3EA45ZCGH5IAZAZIXZ6GVW6SYAOPN3WDX4DLMAESZIRXXVZ65QA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ty5LHXwLmHC7nRPmVfj2wVEif6JQAQQ5Z5218vv1fKK5sD3zE): protocol: write tcp 188.166.225.170:54838->145.79.227.2:4118: use of closed network connection; write tcp 188.166.225.170:54838->145.79.227.2:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54838->145.79.227.2:4118: use of closed network connection Jan 1 03:39:17.912557115 (636h10m17.783640151s) [noise=true , address=145.79.227.2:4118 , piece_id=W3EA45ZCGH5IAZAZIXZ6GVW6SYAOPN3WDX4DLMAESZIRXXVZ65QA, seg_pos={0 0}, upload=557441]: Long tail cancellation due to optimal threshold Jan 1 03:39:17.912582464 (636h10m17.783665499s) [seg_pos={0 0}, upload=557441]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 03:39:17.912645484 (636h10m17.783728519s) [upload=557441]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:39:17.912649532 (636h10m17.783732567s) [upload=557441]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:39:18.208729342 (636h10m18.079812378s) [upload=557441]: Done uploading segments... Jan 1 03:39:18.208744913 (636h10m18.079827958s) [upload=557441]: Upload finished. err: <nil> Duration: 35.947426428s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.............X......X............................X........X.] duration: 3.626487919s trace: https://jaeger.dev.storj.io/trace/8b9520ee0ddc710 FAIL (2026-01-01 03:40:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.626505764s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X....X........................................X] duration: 1m0.003113515s trace: https://jaeger.dev.storj.io/trace/1ebdd84a47b0fea8 FAIL (2026-01-01 03:40:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3 , piece_id=GP4LIOS53UP3P4JAJSMKZRENL66UKKAYPG5Z5LN6AQJQM7RKFPWQ, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.555732730 (636h10m14.426815775s) [noise=true , address=45.91.116.8:4068 , piece_id=2EJJOJHC2EVMRLNTSFHTCCWHUACTM3UCI7557BLV6AGCIKZAIM3Q, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.555861985 (636h10m14.42694503s) [noise=true , address=145.79.227.34:4934 , piece_id=RPORZ7NEJGQK7WXN5PSBXHLYPIM3LAYN4R6GI6L4TZCRDTPTOWCA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.556167417 (636h10m14.427250462s) [noise=true , address=145.79.227.33:4874 , piece_id=ISLR7HHUN5C4JCKEMKIXFJ7X53SK5JJFHAG7SQ5UCIHZMX7FZV6A, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.557132745 (636h10m14.42821578s) [noise=true , address=145.79.227.4:4215 , piece_id=5ZN6S74EDURKEE4QZ7FEOSVOY3NIGK45JUHBJMNGLDWTQUVYQHHA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.557290145 (636h10m14.42837318s) [noise=true , address=64.49.13.169:4710 , piece_id=XOLOSFJMCEWHMYC46T5BD3RUSDZ77AFPFADVEQCDWGRV2JCKGINQ, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.558546016 (636h10m14.429629051s) [noise=true , address=145.79.227.30:4779 , piece_id=IP2UMBX5PXIDW5AOWSZYLD42TKJSWPHDZQYVCH2AZ4LHGXAAWLXA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.559810844 (636h10m14.430893889s) [noise=true , address=145.79.227.44:4203 , piece_id=UCRXAICCLNAQJGY4DWPEJSYG3Q2YU2UNJSIMTGQ6VQFVW2BJ5A2Q, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.561308566 (636h10m14.432391601s) [noise=true , address=145.79.227.10:4421 , piece_id=3SDLFTSJH5W332QOTMC345XUN25LYPBP6HMKTY5ENOEO2WZEGPUA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.562571591 (636h10m14.433654636s) [noise=true , address=145.79.227.43:4164 , piece_id=4J7RF32DYWIYH7NTAPA7WNNXBXQS5B6JOXSDUTLLCMWDDR6IN7OQ, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.563473288 (636h10m14.434556333s) [noise=true , address=145.79.227.40:4053 , piece_id=F73PCPUM44L7KECQWIOKZIX67AVART4WFDM63UEUDV7F3Q66YOJA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.564781790 (636h10m14.435864825s) [noise=true , address=77.90.31.10:4546 , piece_id=OEVEQHYJBGE2FIABEPUYMSOS2NBAM6ZRPQ4J5PAQR3G6QRPS43VA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.565835766 (636h10m14.436918801s) [noise=true , address=145.79.227.41:4107 , piece_id=POHERAN6H5M3HCP5YGK4IKFPUV3JII5QV5QZA6CQRD77XEFGNFFQ, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.567015342 (636h10m14.438098387s) [noise=true , address=145.79.227.2:4118 , piece_id=W3EA45ZCGH5IAZAZIXZ6GVW6SYAOPN3WDX4DLMAESZIRXXVZ65QA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:14.567628530 (636h10m14.438711565s) [noise=true , address=77.90.31.13:4654 , piece_id=ND5P6ZAMBBNT6OFNRIAUORAVPMC45CB6COFGZNZV4WCJKNDKSZBA, seg_pos={0 0}, upload=557441]: Uploading piece... Jan 1 03:39:16.014065836 (636h10m15.885148871s) [upload=557441]: Next returned nil segment Jan 1 03:39:16.014083580 (636h10m15.885166615s) [upload=557441]: Waiting for error group managing segments... Jan 1 03:39:16.455400629 (636h10m16.326483664s) [noise=true , address=145.79.227.10:4421 , piece_id=3SDLFTSJH5W332QOTMC345XUN25LYPBP6HMKTY5ENOEO2WZEGPUA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:16.464872484 (636h10m16.335955529s) [noise=true , address=145.79.227.4:4215 , piece_id=5ZN6S74EDURKEE4QZ7FEOSVOY3NIGK45JUHBJMNGLDWTQUVYQHHA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:16.616875776 (636h10m16.487958821s) [noise=true , address=77.90.31.19:4876 , piece_id=UVRE5SNVM7SVUWD37QXCFDL7FWRQIMGFBYKLUULJKQJ4SW6224PA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:16.759240917 (636h10m16.630323952s) [noise=true , address=77.90.31.21:4907 , piece_id=XELNGQJ3EHMREVTXIFRR4FLXDD5U6CMAQ53GRSJ4WOT76QOQCBGQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:16.826797803 (636h10m16.697880848s) [noise=true , address=77.90.31.10:4546 , piece_id=OEVEQHYJBGE2FIABEPUYMSOS2NBAM6ZRPQ4J5PAQR3G6QRPS43VA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.039826124 (636h10m16.910909169s) [noise=true , address=145.79.227.49:4357 , piece_id=NRGFOQVKNCENLBXQDAU63ZQTJXYDJZ5YMEMZUHNBAWCERZCDI5EQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.319810887 (636h10m17.190893922s) [noise=true , address=145.79.227.30:4779 , piece_id=IP2UMBX5PXIDW5AOWSZYLD42TKJSWPHDZQYVCH2AZ4LHGXAAWLXA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.335275864 (636h10m17.206358899s) [noise=true , address=145.79.227.34:4934 , piece_id=RPORZ7NEJGQK7WXN5PSBXHLYPIM3LAYN4R6GI6L4TZCRDTPTOWCA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.360870549 (636h10m17.231953594s) [noise=true , address=145.79.227.40:4053 , piece_id=F73PCPUM44L7KECQWIOKZIX67AVART4WFDM63UEUDV7F3Q66YOJA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.412439583 (636h10m17.283522628s) [noise=true , address=145.79.227.35:4513 , piece_id=GP4LIOS53UP3P4JAJSMKZRENL66UKKAYPG5Z5LN6AQJQM7RKFPWQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.469220361 (636h10m17.340303396s) [noise=true , address=145.79.227.44:4203 , piece_id=UCRXAICCLNAQJGY4DWPEJSYG3Q2YU2UNJSIMTGQ6VQFVW2BJ5A2Q, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.485589420 (636h10m17.356672455s) [noise=true , address=145.79.227.33:4874 , piece_id=ISLR7HHUN5C4JCKEMKIXFJ7X53SK5JJFHAG7SQ5UCIHZMX7FZV6A, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.598713200 (636h10m17.469796245s) [noise=true , address=145.79.227.43:4164 , piece_id=4J7RF32DYWIYH7NTAPA7WNNXBXQS5B6JOXSDUTLLCMWDDR6IN7OQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.911962313 (636h10m17.783045348s) [noise=true , address=145.79.227.41:4107 , piece_id=POHERAN6H5M3HCP5YGK4IKFPUV3JII5QV5QZA6CQRD77XEFGNFFQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: <nil> Jan 1 03:39:17.911978394 (636h10m17.783061429s) [seg_pos={0 0}, upload=557441]: Segment reached optimal threshold of 14 pieces. Jan 1 03:39:17.912112600 (636h10m17.783195635s) [noise=true , address=77.90.31.13:4654 , piece_id=ND5P6ZAMBBNT6OFNRIAUORAVPMC45CB6COFGZNZV4WCJKNDKSZBA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: expected piece hash; context canceled; EOF Jan 1 03:39:17.912118220 (636h10m17.783201255s) [noise=true , address=77.90.31.13:4654 , piece_id=ND5P6ZAMBBNT6OFNRIAUORAVPMC45CB6COFGZNZV4WCJKNDKSZBA, seg_pos={0 0}, upload=557441]: Long tail cancellation due to optimal threshold Jan 1 03:39:17.912348508 (636h10m17.783431543s) [noise=true , address=45.91.116.8:4068 , piece_id=2EJJOJHC2EVMRLNTSFHTCCWHUACTM3UCI7557BLV6AGCIKZAIM3Q, seg_pos={0 0}, upload=557441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jN6HbQbPBjk645swQd33LrTfb3LbAWk1dPNXU45zFrgmm6Q82): protocol: expected piece hash; context canceled; EOF Jan 1 03:39:17.912354901 (636h10m17.783437936s) [noise=true , address=45.91.116.8:4068 , piece_id=2EJJOJHC2EVMRLNTSFHTCCWHUACTM3UCI7557BLV6AGCIKZAIM3Q, seg_pos={0 0}, upload=557441]: Long tail cancellation due to optimal threshold Jan 1 03:39:17.912458238 (636h10m17.783541283s) [noise=true , address=64.49.13.169:4710 , piece_id=XOLOSFJMCEWHMYC46T5BD3RUSDZ77AFPFADVEQCDWGRV2JCKGINQ, seg_pos={0 0}, upload=557441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:55400->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:55400->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55400->64.49.13.169:4710: use of closed network connection Jan 1 03:39:17.912463217 (636h10m17.783546252s) [noise=true , address=64.49.13.169:4710 , piece_id=XOLOSFJMCEWHMYC46T5BD3RUSDZ77AFPFADVEQCDWGRV2JCKGINQ, seg_pos={0 0}, upload=557441]: Long tail cancellation due to optimal threshold Jan 1 03:39:17.912551856 (636h10m17.783634891s) [noise=true , address=145.79.227.2:4118 , piece_id=W3EA45ZCGH5IAZAZIXZ6GVW6SYAOPN3WDX4DLMAESZIRXXVZ65QA, seg_pos={0 0}, upload=557441]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ty5LHXwLmHC7nRPmVfj2wVEif6JQAQQ5Z5218vv1fKK5sD3zE): protocol: write tcp 188.166.225.170:54838->145.79.227.2:4118: use of closed network connection; write tcp 188.166.225.170:54838->145.79.227.2:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54838->145.79.227.2:4118: use of closed network connection Jan 1 03:39:17.912557115 (636h10m17.783640151s) [noise=true , address=145.79.227.2:4118 , piece_id=W3EA45ZCGH5IAZAZIXZ6GVW6SYAOPN3WDX4DLMAESZIRXXVZ65QA, seg_pos={0 0}, upload=557441]: Long tail cancellation due to optimal threshold Jan 1 03:39:17.912582464 (636h10m17.783665499s) [seg_pos={0 0}, upload=557441]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 03:39:17.912645484 (636h10m17.783728519s) [upload=557441]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:39:17.912649532 (636h10m17.783732567s) [upload=557441]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:39:18.208729342 (636h10m18.079812378s) [upload=557441]: Done uploading segments... Jan 1 03:39:18.208744913 (636h10m18.079827958s) [upload=557441]: Upload finished. err: <nil> Duration: 1m0.003136097s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [..X...X..X..X...X..X....XXX.X.X..XX.X.XXX..XX..XXXXX....XX..] duration: 6.072242599s trace: https://jaeger.dev.storj.io/trace/3a1918a7502a0c40 FAIL (2026-01-01 03:39:57 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 1 03:38:18.274492318 (636h9m18.145575353s) [upload=557425]: Starting upload Jan 1 03:38:18.274567070 (636h9m18.145650105s) [upload=557425]: Done starting upload Jan 1 03:38:18.274688662 (636h9m18.145771707s) [upload=557425]: Uploading segments... Jan 1 03:38:18.274806416 (636h9m18.145889461s) [upload=557425]: Got next segment. Inline: false Jan 1 03:38:18.274842244 (636h9m18.145925289s) [upload=557425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:38:18.274847464 (636h9m18.145930519s) [upload=557425]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:38:18.522138205 (636h9m18.39322127s) [seg_pos={0 0}, upload=557425]: Begin upload segment... Jan 1 03:38:18.522195464 (636h9m18.393278509s) [seg_pos={0 0}, upload=557425]: Done begin upload segment. Jan 1 03:38:18.522222756 (636h9m18.393305791s) [noise=true , address=145.79.227.15:4874 , piece_id=LCSGCVZS3FMNO3EG45DQRE4WIB6356TDEYCAM7SFEQIRGNMMZHVQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.522240590 (636h9m18.393323655s) [noise=true , address=145.79.227.5:4221 , piece_id=LR5HLZ2FXOW2VBFYUH7N4NCEX46VWHJIKXXT7VQCOXH3R2PY5H6Q, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.522841954 (636h9m18.393924999s) [noise=true , address=45.91.116.6:4976 , piece_id=5RDOOLXQNMLR2Z5XSV4YI42ZBKPPPHLIFDZ6WBUG6T5CKGSFH3BQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523203123 (636h9m18.394286168s) [noise=true , address=45.91.116.8:4046 , piece_id=AATWWQ44SIMWQETNR37O7OSCOQFOIRSQDW7QTVJOTM75FBD2TGXQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523238200 (636h9m18.394321245s) [noise=true , address=145.79.227.19:4077 , piece_id=4R4DDZL5TEPSTNNLYMWZYPWJX62WM3JKV4UHBQYQQIJMEOS4KR6A, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523581053 (636h9m18.394664088s) [noise=true , address=145.79.227.37:4567 , piece_id=6UKCBEODRCAEAWW2UXHFQHVUBLBTFEGFPJXZH27EFBKALQCS34NA, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523675452 (636h9m18.394758487s) [noise=true , address=145.79.227.47:4266 , piece_id=BYNWCGK4PXBHSE4SNQUT24MUOPBXUFIY225736NRWKNLXEKJ5LDA, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523829435 (636h9m18.39491247s) [noise=true , address=145.79.227.17:4970 , piece_id=4QCPG54UE4X3CXT6H2UOQPEO4WHGMVZM37V6YFN5RDCKRGAYAGDQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523980252 (636h9m18.395063287s) [noise=true , address=145.79.227.22:4509 , piece_id=I4F3MQKLEZRE2YC37V2TQQO7E2XNOPTWKLHEDLYXGPJE2NVQCNEQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524079642 (636h9m18.395162677s) [noise=true , address=145.79.227.18:4007 , piece_id=2A2MEUNPIIQR4FBUEVGOY4HO5MGV6LAOFODMEDEO6OQRUVLK62AA, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524396465 (636h9m18.3954795s) [noise=true , address=145.79.227.29:4942 , piece_id=ZJEEMQFL7RKIOB3OILJD47A6LHRDVTLK66X34KVFQ46H7RU7MPSQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524481868 (636h9m18.395564943s) [noise=true , address=77.90.31.12:4589 , piece_id=FGSJYHVF6QRKWSF6DOP7DMXMRF64BBMIBVX7VPPMFXJOFSQ3C47A, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524653404 (636h9m18.395736439s) [noise=true , address=45.91.116.10:4083 , piece_id=QITJ4BWAMDX5G2M5R5HTTS5A3ZHWCJQI56HM66CRSW4IP4PAJKNQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524906376 (636h9m18.395989411s) [noise=true , address=145.79.227.36:4519 , piece_id=IEXN5NPPGDTELLYLLZYMCUYENTIN6A4N4ALEPJIEHSF4INKNTPAQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524912417 (636h9m18.395995452s) [noise=true , address=145.79.227.11:4731 , piece_id=MXYDHPFONCMBGPUXEO2DRLOAZQ45VRFLCZP6MXBQFEUAHVS2HEQQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.525774960 (636h9m18.396857995s) [noise=true , address=145.79.227.46:4601 , piece_id=R2ZUOVU3KQIVFEYWJLTX6BBZVPEFKE5LMSDTWGGOZDTEW2NORMKQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:20.226070182 (636h9m20.097153217s) [upload=557425]: Next returned nil segment Jan 1 03:38:20.226084649 (636h9m20.097167684s) [upload=557425]: Waiting for error group managing segments... Jan 1 03:38:20.500808796 (636h9m20.371891841s) [noise=true , address=145.79.227.11:4731 , piece_id=MXYDHPFONCMBGPUXEO2DRLOAZQ45VRFLCZP6MXBQFEUAHVS2HEQQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.511146620 (636h9m20.382229665s) [noise=true , address=145.79.227.46:4601 , piece_id=R2ZUOVU3KQIVFEYWJLTX6BBZVPEFKE5LMSDTWGGOZDTEW2NORMKQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.512963139 (636h9m20.384046184s) [noise=true , address=145.79.227.5:4221 , piece_id=LR5HLZ2FXOW2VBFYUH7N4NCEX46VWHJIKXXT7VQCOXH3R2PY5H6Q, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.513585224 (636h9m20.384668269s) [noise=true , address=145.79.227.15:4874 , piece_id=LCSGCVZS3FMNO3EG45DQRE4WIB6356TDEYCAM7SFEQIRGNMMZHVQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.517155291 (636h9m20.388238336s) [noise=true , address=145.79.227.17:4970 , piece_id=4QCPG54UE4X3CXT6H2UOQPEO4WHGMVZM37V6YFN5RDCKRGAYAGDQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.528961282 (636h9m20.400044317s) [noise=true , address=145.79.227.19:4077 , piece_id=4R4DDZL5TEPSTNNLYMWZYPWJX62WM3JKV4UHBQYQQIJMEOS4KR6A, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.530863914 (636h9m20.401946949s) [noise=true , address=145.79.227.22:4509 , piece_id=I4F3MQKLEZRE2YC37V2TQQO7E2XNOPTWKLHEDLYXGPJE2NVQCNEQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.538983445 (636h9m20.41006649s) [noise=true , address=145.79.227.18:4007 , piece_id=2A2MEUNPIIQR4FBUEVGOY4HO5MGV6LAOFODMEDEO6OQRUVLK62AA, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.595792125 (636h9m20.46687516s) [noise=true , address=145.79.227.36:4519 , piece_id=IEXN5NPPGDTELLYLLZYMCUYENTIN6A4N4ALEPJIEHSF4INKNTPAQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.612204215 (636h9m20.48328725s) [noise=true , address=145.79.227.47:4266 , piece_id=BYNWCGK4PXBHSE4SNQUT24MUOPBXUFIY225736NRWKNLXEKJ5LDA, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.767931659 (636h9m20.639014694s) [noise=true , address=77.90.31.12:4589 , piece_id=FGSJYHVF6QRKWSF6DOP7DMXMRF64BBMIBVX7VPPMFXJOFSQ3C47A, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.867883629 (636h9m20.738966664s) [noise=true , address=45.91.116.10:4083 , piece_id=QITJ4BWAMDX5G2M5R5HTTS5A3ZHWCJQI56HM66CRSW4IP4PAJKNQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.871885079 (636h9m20.742968114s) [noise=true , address=45.91.116.8:4046 , piece_id=AATWWQ44SIMWQETNR37O7OSCOQFOIRSQDW7QTVJOTM75FBD2TGXQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.871902772 (636h9m20.742985807s) [seg_pos={0 0}, upload=557425]: Segment reached optimal threshold of 13 pieces. Jan 1 03:38:20.872093115 (636h9m20.74317618s) [noise=true , address=145.79.227.29:4942 , piece_id=ZJEEMQFL7RKIOB3OILJD47A6LHRDVTLK66X34KVFQ46H7RU7MPSQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Jan 1 03:38:20.872105989 (636h9m20.743189024s) [noise=true , address=145.79.227.29:4942 , piece_id=ZJEEMQFL7RKIOB3OILJD47A6LHRDVTLK66X34KVFQ46H7RU7MPSQ, seg_pos={0 0}, upload=557425]: Long tail cancellation due to optimal threshold Jan 1 03:38:20.872304367 (636h9m20.743387402s) [noise=true , address=45.91.116.6:4976 , piece_id=5RDOOLXQNMLR2Z5XSV4YI42ZBKPPPHLIFDZ6WBUG6T5CKGSFH3BQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 1 03:38:20.872325717 (636h9m20.743408772s) [noise=true , address=45.91.116.6:4976 , piece_id=5RDOOLXQNMLR2Z5XSV4YI42ZBKPPPHLIFDZ6WBUG6T5CKGSFH3BQ, seg_pos={0 0}, upload=557425]: Long tail cancellation due to optimal threshold Jan 1 03:38:20.872342800 (636h9m20.743425946s) [noise=true , address=145.79.227.37:4567 , piece_id=6UKCBEODRCAEAWW2UXHFQHVUBLBTFEGFPJXZH27EFBKALQCS34NA, seg_pos={0 0}, upload=557425]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:50636->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:50636->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50636->145.79.227.37:4567: use of closed network connection Jan 1 03:38:20.872355464 (636h9m20.743438509s) [noise=true , address=145.79.227.37:4567 , piece_id=6UKCBEODRCAEAWW2UXHFQHVUBLBTFEGFPJXZH27EFBKALQCS34NA, seg_pos={0 0}, upload=557425]: Long tail cancellation due to optimal threshold Jan 1 03:38:20.872439024 (636h9m20.743522069s) [seg_pos={0 0}, upload=557425]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:38:20.872567558 (636h9m20.743650603s) [upload=557425]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:38:20.872573520 (636h9m20.743656555s) [upload=557425]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:38:21.172543628 (636h9m21.043626673s) [upload=557425]: Done uploading segments... Jan 1 03:38:21.172563316 (636h9m21.043646361s) [upload=557425]: Upload finished. err: <nil> Duration: 6.07227494s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..............X......X............................X........X] duration: 5.001156187s trace: https://jaeger.dev.storj.io/trace/8b9520ee0ddc710 FAIL (2026-01-01 03:39:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001180133s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [...X...X..X..X...X..X....XXX.X.X..XX.X.XXX..XX..XXXXX....XX.] duration: 5.761767777s trace: https://jaeger.dev.storj.io/trace/3a1918a7502a0c40 FAIL (2026-01-01 03:39:27 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 1 03:38:18.274492318 (636h9m18.145575353s) [upload=557425]: Starting upload Jan 1 03:38:18.274567070 (636h9m18.145650105s) [upload=557425]: Done starting upload Jan 1 03:38:18.274688662 (636h9m18.145771707s) [upload=557425]: Uploading segments... Jan 1 03:38:18.274806416 (636h9m18.145889461s) [upload=557425]: Got next segment. Inline: false Jan 1 03:38:18.274842244 (636h9m18.145925289s) [upload=557425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:38:18.274847464 (636h9m18.145930519s) [upload=557425]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:38:18.522138205 (636h9m18.39322127s) [seg_pos={0 0}, upload=557425]: Begin upload segment... Jan 1 03:38:18.522195464 (636h9m18.393278509s) [seg_pos={0 0}, upload=557425]: Done begin upload segment. Jan 1 03:38:18.522222756 (636h9m18.393305791s) [noise=true , address=145.79.227.15:4874 , piece_id=LCSGCVZS3FMNO3EG45DQRE4WIB6356TDEYCAM7SFEQIRGNMMZHVQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.522240590 (636h9m18.393323655s) [noise=true , address=145.79.227.5:4221 , piece_id=LR5HLZ2FXOW2VBFYUH7N4NCEX46VWHJIKXXT7VQCOXH3R2PY5H6Q, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.522841954 (636h9m18.393924999s) [noise=true , address=45.91.116.6:4976 , piece_id=5RDOOLXQNMLR2Z5XSV4YI42ZBKPPPHLIFDZ6WBUG6T5CKGSFH3BQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523203123 (636h9m18.394286168s) [noise=true , address=45.91.116.8:4046 , piece_id=AATWWQ44SIMWQETNR37O7OSCOQFOIRSQDW7QTVJOTM75FBD2TGXQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523238200 (636h9m18.394321245s) [noise=true , address=145.79.227.19:4077 , piece_id=4R4DDZL5TEPSTNNLYMWZYPWJX62WM3JKV4UHBQYQQIJMEOS4KR6A, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523581053 (636h9m18.394664088s) [noise=true , address=145.79.227.37:4567 , piece_id=6UKCBEODRCAEAWW2UXHFQHVUBLBTFEGFPJXZH27EFBKALQCS34NA, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523675452 (636h9m18.394758487s) [noise=true , address=145.79.227.47:4266 , piece_id=BYNWCGK4PXBHSE4SNQUT24MUOPBXUFIY225736NRWKNLXEKJ5LDA, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523829435 (636h9m18.39491247s) [noise=true , address=145.79.227.17:4970 , piece_id=4QCPG54UE4X3CXT6H2UOQPEO4WHGMVZM37V6YFN5RDCKRGAYAGDQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523980252 (636h9m18.395063287s) [noise=true , address=145.79.227.22:4509 , piece_id=I4F3MQKLEZRE2YC37V2TQQO7E2XNOPTWKLHEDLYXGPJE2NVQCNEQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524079642 (636h9m18.395162677s) [noise=true , address=145.79.227.18:4007 , piece_id=2A2MEUNPIIQR4FBUEVGOY4HO5MGV6LAOFODMEDEO6OQRUVLK62AA, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524396465 (636h9m18.3954795s) [noise=true , address=145.79.227.29:4942 , piece_id=ZJEEMQFL7RKIOB3OILJD47A6LHRDVTLK66X34KVFQ46H7RU7MPSQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524481868 (636h9m18.395564943s) [noise=true , address=77.90.31.12:4589 , piece_id=FGSJYHVF6QRKWSF6DOP7DMXMRF64BBMIBVX7VPPMFXJOFSQ3C47A, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524653404 (636h9m18.395736439s) [noise=true , address=45.91.116.10:4083 , piece_id=QITJ4BWAMDX5G2M5R5HTTS5A3ZHWCJQI56HM66CRSW4IP4PAJKNQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524906376 (636h9m18.395989411s) [noise=true , address=145.79.227.36:4519 , piece_id=IEXN5NPPGDTELLYLLZYMCUYENTIN6A4N4ALEPJIEHSF4INKNTPAQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524912417 (636h9m18.395995452s) [noise=true , address=145.79.227.11:4731 , piece_id=MXYDHPFONCMBGPUXEO2DRLOAZQ45VRFLCZP6MXBQFEUAHVS2HEQQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.525774960 (636h9m18.396857995s) [noise=true , address=145.79.227.46:4601 , piece_id=R2ZUOVU3KQIVFEYWJLTX6BBZVPEFKE5LMSDTWGGOZDTEW2NORMKQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:20.226070182 (636h9m20.097153217s) [upload=557425]: Next returned nil segment Jan 1 03:38:20.226084649 (636h9m20.097167684s) [upload=557425]: Waiting for error group managing segments... Jan 1 03:38:20.500808796 (636h9m20.371891841s) [noise=true , address=145.79.227.11:4731 , piece_id=MXYDHPFONCMBGPUXEO2DRLOAZQ45VRFLCZP6MXBQFEUAHVS2HEQQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.511146620 (636h9m20.382229665s) [noise=true , address=145.79.227.46:4601 , piece_id=R2ZUOVU3KQIVFEYWJLTX6BBZVPEFKE5LMSDTWGGOZDTEW2NORMKQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.512963139 (636h9m20.384046184s) [noise=true , address=145.79.227.5:4221 , piece_id=LR5HLZ2FXOW2VBFYUH7N4NCEX46VWHJIKXXT7VQCOXH3R2PY5H6Q, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.513585224 (636h9m20.384668269s) [noise=true , address=145.79.227.15:4874 , piece_id=LCSGCVZS3FMNO3EG45DQRE4WIB6356TDEYCAM7SFEQIRGNMMZHVQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.517155291 (636h9m20.388238336s) [noise=true , address=145.79.227.17:4970 , piece_id=4QCPG54UE4X3CXT6H2UOQPEO4WHGMVZM37V6YFN5RDCKRGAYAGDQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.528961282 (636h9m20.400044317s) [noise=true , address=145.79.227.19:4077 , piece_id=4R4DDZL5TEPSTNNLYMWZYPWJX62WM3JKV4UHBQYQQIJMEOS4KR6A, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.530863914 (636h9m20.401946949s) [noise=true , address=145.79.227.22:4509 , piece_id=I4F3MQKLEZRE2YC37V2TQQO7E2XNOPTWKLHEDLYXGPJE2NVQCNEQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.538983445 (636h9m20.41006649s) [noise=true , address=145.79.227.18:4007 , piece_id=2A2MEUNPIIQR4FBUEVGOY4HO5MGV6LAOFODMEDEO6OQRUVLK62AA, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.595792125 (636h9m20.46687516s) [noise=true , address=145.79.227.36:4519 , piece_id=IEXN5NPPGDTELLYLLZYMCUYENTIN6A4N4ALEPJIEHSF4INKNTPAQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.612204215 (636h9m20.48328725s) [noise=true , address=145.79.227.47:4266 , piece_id=BYNWCGK4PXBHSE4SNQUT24MUOPBXUFIY225736NRWKNLXEKJ5LDA, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.767931659 (636h9m20.639014694s) [noise=true , address=77.90.31.12:4589 , piece_id=FGSJYHVF6QRKWSF6DOP7DMXMRF64BBMIBVX7VPPMFXJOFSQ3C47A, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.867883629 (636h9m20.738966664s) [noise=true , address=45.91.116.10:4083 , piece_id=QITJ4BWAMDX5G2M5R5HTTS5A3ZHWCJQI56HM66CRSW4IP4PAJKNQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.871885079 (636h9m20.742968114s) [noise=true , address=45.91.116.8:4046 , piece_id=AATWWQ44SIMWQETNR37O7OSCOQFOIRSQDW7QTVJOTM75FBD2TGXQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.871902772 (636h9m20.742985807s) [seg_pos={0 0}, upload=557425]: Segment reached optimal threshold of 13 pieces. Jan 1 03:38:20.872093115 (636h9m20.74317618s) [noise=true , address=145.79.227.29:4942 , piece_id=ZJEEMQFL7RKIOB3OILJD47A6LHRDVTLK66X34KVFQ46H7RU7MPSQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Jan 1 03:38:20.872105989 (636h9m20.743189024s) [noise=true , address=145.79.227.29:4942 , piece_id=ZJEEMQFL7RKIOB3OILJD47A6LHRDVTLK66X34KVFQ46H7RU7MPSQ, seg_pos={0 0}, upload=557425]: Long tail cancellation due to optimal threshold Jan 1 03:38:20.872304367 (636h9m20.743387402s) [noise=true , address=45.91.116.6:4976 , piece_id=5RDOOLXQNMLR2Z5XSV4YI42ZBKPPPHLIFDZ6WBUG6T5CKGSFH3BQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 1 03:38:20.872325717 (636h9m20.743408772s) [noise=true , address=45.91.116.6:4976 , piece_id=5RDOOLXQNMLR2Z5XSV4YI42ZBKPPPHLIFDZ6WBUG6T5CKGSFH3BQ, seg_pos={0 0}, upload=557425]: Long tail cancellation due to optimal threshold Jan 1 03:38:20.872342800 (636h9m20.743425946s) [noise=true , address=145.79.227.37:4567 , piece_id=6UKCBEODRCAEAWW2UXHFQHVUBLBTFEGFPJXZH27EFBKALQCS34NA, seg_pos={0 0}, upload=557425]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:50636->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:50636->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50636->145.79.227.37:4567: use of closed network connection Jan 1 03:38:20.872355464 (636h9m20.743438509s) [noise=true , address=145.79.227.37:4567 , piece_id=6UKCBEODRCAEAWW2UXHFQHVUBLBTFEGFPJXZH27EFBKALQCS34NA, seg_pos={0 0}, upload=557425]: Long tail cancellation due to optimal threshold Jan 1 03:38:20.872439024 (636h9m20.743522069s) [seg_pos={0 0}, upload=557425]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:38:20.872567558 (636h9m20.743650603s) [upload=557425]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:38:20.872573520 (636h9m20.743656555s) [upload=557425]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:38:21.172543628 (636h9m21.043626673s) [upload=557425]: Done uploading segments... Jan 1 03:38:21.172563316 (636h9m21.043646361s) [upload=557425]: Upload finished. err: <nil> Duration: 5.761799509s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X....X........................................] duration: 29.008852313s trace: https://jaeger.dev.storj.io/trace/725fc16e891a4127 FAIL (2026-01-01 03:39:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3:03:23.412681369 (635h34m23.283764404s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413214023 (635h34m23.284297058s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413504245 (635h34m23.28458728s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413821750 (635h34m23.284904785s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414124316 (635h34m23.285207351s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414417554 (635h34m23.285500589s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414689172 (635h34m23.285772207s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415023338 (635h34m23.286106373s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415341384 (635h34m23.286424419s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415659309 (635h34m23.286742354s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416184289 (635h34m23.287267324s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416571025 (635h34m23.28765406s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416998890 (635h34m23.288081925s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.417376770 (635h34m23.288459815s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.418791664 (635h34m23.289874709s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.420183384 (635h34m23.291266419s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.421563200 (635h34m23.292646246s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:24.823751878 (635h34m24.694834913s) [upload=556915]: Next returned nil segment Jan 1 03:03:24.823765594 (635h34m24.694848629s) [upload=556915]: Waiting for error group managing segments... Jan 1 03:03:25.191603537 (635h34m25.062686582s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.271741648 (635h34m25.142824683s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.278781112 (635h34m25.149864157s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.337014705 (635h34m25.20809774s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.372659678 (635h34m25.243742713s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.439670894 (635h34m25.31075394s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.447380494 (635h34m25.318463529s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.459093186 (635h34m25.330176231s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.602592395 (635h34m25.47367543s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.719366504 (635h34m25.59044954s) [noise=true , address=45.91.116.12:4230 , piece_id=WQHO3KOUTZSHBZIDUXANBVFSUKTBKEWHTZ3Z7BNGXGOFP2BMJHWA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.875237800 (635h34m25.746320845s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.143929177 (635h34m26.015012212s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.159675169 (635h34m26.030758204s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181352455 (635h34m26.05243549s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181379146 (635h34m26.052462181s) [seg_pos={0 0}, upload=556915]: Segment reached optimal threshold of 14 pieces. Jan 1 03:03:26.181716378 (635h34m26.052799413s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Pvm1SftEF3tnCEBqUn1ZmyVcLDBkR6pyegEb7mtzjdQiiqYqt): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181723281 (635h34m26.052806316s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181784638 (635h34m26.052867703s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181848530 (635h34m26.052931565s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181859921 (635h34m26.052942956s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181866474 (635h34m26.052949509s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181920486 (635h34m26.053003522s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:184g6qHdezLW3d2CKNW8GRgdzeVLQuMEhd8B8c1zEnkKNkeSTu): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181925296 (635h34m26.053008331s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181941377 (635h34m26.053024412s) [seg_pos={0 0}, upload=556915]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 03:03:26.182003695 (635h34m26.05308673s) [upload=556915]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:03:26.182009416 (635h34m26.053092461s) [upload=556915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:03:26.474026257 (635h34m26.345109292s) [upload=556915]: Done uploading segments... Jan 1 03:03:26.474049391 (635h34m26.345132427s) [upload=556915]: Upload finished. err: <nil> Duration: 29.008887641s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...............X......X............................X........] duration: 3.597429598s trace: https://jaeger.dev.storj.io/trace/2ad779a295dfac2 FAIL (2026-01-01 03:38:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.597461187s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [....X...X..X..X...X..X....XXX.X.X..XX.X.XXX..XX..XXXXX....XX] duration: 30.004150741s trace: https://jaeger.dev.storj.io/trace/3a1918a7502a0c40 FAIL (2026-01-01 03:38:47 UTC) orbiter warning: operation: DeleteObject ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 1 03:38:18.274492318 (636h9m18.145575353s) [upload=557425]: Starting upload Jan 1 03:38:18.274567070 (636h9m18.145650105s) [upload=557425]: Done starting upload Jan 1 03:38:18.274688662 (636h9m18.145771707s) [upload=557425]: Uploading segments... Jan 1 03:38:18.274806416 (636h9m18.145889461s) [upload=557425]: Got next segment. Inline: false Jan 1 03:38:18.274842244 (636h9m18.145925289s) [upload=557425]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:38:18.274847464 (636h9m18.145930519s) [upload=557425]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:38:18.522138205 (636h9m18.39322127s) [seg_pos={0 0}, upload=557425]: Begin upload segment... Jan 1 03:38:18.522195464 (636h9m18.393278509s) [seg_pos={0 0}, upload=557425]: Done begin upload segment. Jan 1 03:38:18.522222756 (636h9m18.393305791s) [noise=true , address=145.79.227.15:4874 , piece_id=LCSGCVZS3FMNO3EG45DQRE4WIB6356TDEYCAM7SFEQIRGNMMZHVQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.522240590 (636h9m18.393323655s) [noise=true , address=145.79.227.5:4221 , piece_id=LR5HLZ2FXOW2VBFYUH7N4NCEX46VWHJIKXXT7VQCOXH3R2PY5H6Q, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.522841954 (636h9m18.393924999s) [noise=true , address=45.91.116.6:4976 , piece_id=5RDOOLXQNMLR2Z5XSV4YI42ZBKPPPHLIFDZ6WBUG6T5CKGSFH3BQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523203123 (636h9m18.394286168s) [noise=true , address=45.91.116.8:4046 , piece_id=AATWWQ44SIMWQETNR37O7OSCOQFOIRSQDW7QTVJOTM75FBD2TGXQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523238200 (636h9m18.394321245s) [noise=true , address=145.79.227.19:4077 , piece_id=4R4DDZL5TEPSTNNLYMWZYPWJX62WM3JKV4UHBQYQQIJMEOS4KR6A, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523581053 (636h9m18.394664088s) [noise=true , address=145.79.227.37:4567 , piece_id=6UKCBEODRCAEAWW2UXHFQHVUBLBTFEGFPJXZH27EFBKALQCS34NA, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523675452 (636h9m18.394758487s) [noise=true , address=145.79.227.47:4266 , piece_id=BYNWCGK4PXBHSE4SNQUT24MUOPBXUFIY225736NRWKNLXEKJ5LDA, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523829435 (636h9m18.39491247s) [noise=true , address=145.79.227.17:4970 , piece_id=4QCPG54UE4X3CXT6H2UOQPEO4WHGMVZM37V6YFN5RDCKRGAYAGDQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.523980252 (636h9m18.395063287s) [noise=true , address=145.79.227.22:4509 , piece_id=I4F3MQKLEZRE2YC37V2TQQO7E2XNOPTWKLHEDLYXGPJE2NVQCNEQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524079642 (636h9m18.395162677s) [noise=true , address=145.79.227.18:4007 , piece_id=2A2MEUNPIIQR4FBUEVGOY4HO5MGV6LAOFODMEDEO6OQRUVLK62AA, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524396465 (636h9m18.3954795s) [noise=true , address=145.79.227.29:4942 , piece_id=ZJEEMQFL7RKIOB3OILJD47A6LHRDVTLK66X34KVFQ46H7RU7MPSQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524481868 (636h9m18.395564943s) [noise=true , address=77.90.31.12:4589 , piece_id=FGSJYHVF6QRKWSF6DOP7DMXMRF64BBMIBVX7VPPMFXJOFSQ3C47A, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524653404 (636h9m18.395736439s) [noise=true , address=45.91.116.10:4083 , piece_id=QITJ4BWAMDX5G2M5R5HTTS5A3ZHWCJQI56HM66CRSW4IP4PAJKNQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524906376 (636h9m18.395989411s) [noise=true , address=145.79.227.36:4519 , piece_id=IEXN5NPPGDTELLYLLZYMCUYENTIN6A4N4ALEPJIEHSF4INKNTPAQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.524912417 (636h9m18.395995452s) [noise=true , address=145.79.227.11:4731 , piece_id=MXYDHPFONCMBGPUXEO2DRLOAZQ45VRFLCZP6MXBQFEUAHVS2HEQQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:18.525774960 (636h9m18.396857995s) [noise=true , address=145.79.227.46:4601 , piece_id=R2ZUOVU3KQIVFEYWJLTX6BBZVPEFKE5LMSDTWGGOZDTEW2NORMKQ, seg_pos={0 0}, upload=557425]: Uploading piece... Jan 1 03:38:20.226070182 (636h9m20.097153217s) [upload=557425]: Next returned nil segment Jan 1 03:38:20.226084649 (636h9m20.097167684s) [upload=557425]: Waiting for error group managing segments... Jan 1 03:38:20.500808796 (636h9m20.371891841s) [noise=true , address=145.79.227.11:4731 , piece_id=MXYDHPFONCMBGPUXEO2DRLOAZQ45VRFLCZP6MXBQFEUAHVS2HEQQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.511146620 (636h9m20.382229665s) [noise=true , address=145.79.227.46:4601 , piece_id=R2ZUOVU3KQIVFEYWJLTX6BBZVPEFKE5LMSDTWGGOZDTEW2NORMKQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.512963139 (636h9m20.384046184s) [noise=true , address=145.79.227.5:4221 , piece_id=LR5HLZ2FXOW2VBFYUH7N4NCEX46VWHJIKXXT7VQCOXH3R2PY5H6Q, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.513585224 (636h9m20.384668269s) [noise=true , address=145.79.227.15:4874 , piece_id=LCSGCVZS3FMNO3EG45DQRE4WIB6356TDEYCAM7SFEQIRGNMMZHVQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.517155291 (636h9m20.388238336s) [noise=true , address=145.79.227.17:4970 , piece_id=4QCPG54UE4X3CXT6H2UOQPEO4WHGMVZM37V6YFN5RDCKRGAYAGDQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.528961282 (636h9m20.400044317s) [noise=true , address=145.79.227.19:4077 , piece_id=4R4DDZL5TEPSTNNLYMWZYPWJX62WM3JKV4UHBQYQQIJMEOS4KR6A, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.530863914 (636h9m20.401946949s) [noise=true , address=145.79.227.22:4509 , piece_id=I4F3MQKLEZRE2YC37V2TQQO7E2XNOPTWKLHEDLYXGPJE2NVQCNEQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.538983445 (636h9m20.41006649s) [noise=true , address=145.79.227.18:4007 , piece_id=2A2MEUNPIIQR4FBUEVGOY4HO5MGV6LAOFODMEDEO6OQRUVLK62AA, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.595792125 (636h9m20.46687516s) [noise=true , address=145.79.227.36:4519 , piece_id=IEXN5NPPGDTELLYLLZYMCUYENTIN6A4N4ALEPJIEHSF4INKNTPAQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.612204215 (636h9m20.48328725s) [noise=true , address=145.79.227.47:4266 , piece_id=BYNWCGK4PXBHSE4SNQUT24MUOPBXUFIY225736NRWKNLXEKJ5LDA, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.767931659 (636h9m20.639014694s) [noise=true , address=77.90.31.12:4589 , piece_id=FGSJYHVF6QRKWSF6DOP7DMXMRF64BBMIBVX7VPPMFXJOFSQ3C47A, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.867883629 (636h9m20.738966664s) [noise=true , address=45.91.116.10:4083 , piece_id=QITJ4BWAMDX5G2M5R5HTTS5A3ZHWCJQI56HM66CRSW4IP4PAJKNQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.871885079 (636h9m20.742968114s) [noise=true , address=45.91.116.8:4046 , piece_id=AATWWQ44SIMWQETNR37O7OSCOQFOIRSQDW7QTVJOTM75FBD2TGXQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: <nil> Jan 1 03:38:20.871902772 (636h9m20.742985807s) [seg_pos={0 0}, upload=557425]: Segment reached optimal threshold of 13 pieces. Jan 1 03:38:20.872093115 (636h9m20.74317618s) [noise=true , address=145.79.227.29:4942 , piece_id=ZJEEMQFL7RKIOB3OILJD47A6LHRDVTLK66X34KVFQ46H7RU7MPSQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF Jan 1 03:38:20.872105989 (636h9m20.743189024s) [noise=true , address=145.79.227.29:4942 , piece_id=ZJEEMQFL7RKIOB3OILJD47A6LHRDVTLK66X34KVFQ46H7RU7MPSQ, seg_pos={0 0}, upload=557425]: Long tail cancellation due to optimal threshold Jan 1 03:38:20.872304367 (636h9m20.743387402s) [noise=true , address=45.91.116.6:4976 , piece_id=5RDOOLXQNMLR2Z5XSV4YI42ZBKPPPHLIFDZ6WBUG6T5CKGSFH3BQ, seg_pos={0 0}, upload=557425]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Jan 1 03:38:20.872325717 (636h9m20.743408772s) [noise=true , address=45.91.116.6:4976 , piece_id=5RDOOLXQNMLR2Z5XSV4YI42ZBKPPPHLIFDZ6WBUG6T5CKGSFH3BQ, seg_pos={0 0}, upload=557425]: Long tail cancellation due to optimal threshold Jan 1 03:38:20.872342800 (636h9m20.743425946s) [noise=true , address=145.79.227.37:4567 , piece_id=6UKCBEODRCAEAWW2UXHFQHVUBLBTFEGFPJXZH27EFBKALQCS34NA, seg_pos={0 0}, upload=557425]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19uEj8XgtM7PdNP8KosTCF6H1dSrJ25x6fMshJHQbyrHzJ5sDS): protocol: write tcp 188.166.225.170:50636->145.79.227.37:4567: use of closed network connection; write tcp 188.166.225.170:50636->145.79.227.37:4567: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50636->145.79.227.37:4567: use of closed network connection Jan 1 03:38:20.872355464 (636h9m20.743438509s) [noise=true , address=145.79.227.37:4567 , piece_id=6UKCBEODRCAEAWW2UXHFQHVUBLBTFEGFPJXZH27EFBKALQCS34NA, seg_pos={0 0}, upload=557425]: Long tail cancellation due to optimal threshold Jan 1 03:38:20.872439024 (636h9m20.743522069s) [seg_pos={0 0}, upload=557425]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:38:20.872567558 (636h9m20.743650603s) [upload=557425]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:38:20.872573520 (636h9m20.743656555s) [upload=557425]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:38:21.172543628 (636h9m21.043626673s) [upload=557425]: Done uploading segments... Jan 1 03:38:21.172563316 (636h9m21.043646361s) [upload=557425]: Upload finished. err: <nil> Duration: 30.004172933s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [................X......X............................X.......] duration: 3.520895048s trace: https://jaeger.dev.storj.io/trace/2ad779a295dfac2 FAIL (2026-01-01 03:38:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.520918812s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.....X...X..X..X...X..X....XXX.X.X..XX.X.XXX..XX..XXXXX....X] duration: 30.003116979s trace: https://jaeger.dev.storj.io/trace/549f4351e4519740 FAIL (2026-01-01 03:38:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 (636h8m40.904428876s) [upload=557417]: Uploading segments... Jan 1 03:37:41.033431844 (636h8m40.904514889s) [upload=557417]: Got next segment. Inline: false Jan 1 03:37:41.033484755 (636h8m40.904567801s) [upload=557417]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:37:41.033490837 (636h8m40.904573882s) [upload=557417]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:37:41.280960819 (636h8m41.152043854s) [seg_pos={0 0}, upload=557417]: Begin upload segment... Jan 1 03:37:41.281134160 (636h8m41.152217205s) [seg_pos={0 0}, upload=557417]: Done begin upload segment. Jan 1 03:37:41.281167002 (636h8m41.152250037s) [noise=true , address=145.79.227.8:4331 , piece_id=JJVBSDXEGXOQ7N3GLYGU4BPRFS2FWLD7DHHPZ5EXNIUF2ATYSJKQ, seg_pos={0 0}, upload=557417]: Uploading piece... Jan 1 03:37:41.281258205 (636h8m41.15234125s) [noise=true , address=77.90.31.18:4805 , piece_id=YHNGUE42NSLAIOMYWQ4SJ2C7O4BUKUAEXTYEIJZYCKKZCBMOSFQQ, seg_pos={0 0}, upload=557417]: Uploading piece... Jan 1 03:37:41.281703664 (636h8m41.152786699s) [noise=true , address=145.79.227.41:4087 , piece_id=7SV7CLVJGNZQJNQJPXN2NBDPOQ24QYN654PX6MFTJ62IVGI27BOQ, seg_pos={0 0}, upload=557417]: Uploading piece... Jan 1 03:37:41.282189188 (636h8m41.153272223s) [noise=true , address=145.79.227.31:4797 , piece_id=QEXJ5KUPNJZ5PAISMBNMPCHQL2ZQL5IEMRESHOIBOYMWG22NGVLQ, seg_pos={0 0}, upload=557417]: Uploading piece... Jan 1 03:37:41.282690502 (636h8m41.153773547s) [noise=true , address=45.91.116.4:4940 , piece_id=DVLVAHFQIIIJLUV5AC6X6CNPULNVLFQBTJ5BNUP2SMJV5Y2MSZJA, seg_pos={0 0}, upload=557417]: Uploading piece... Jan 1 03:37:41.282817735 (636h8m41.15390077s) [noise=true , address=145.79.227.39:4013 , piece_id=LYFILPOG4UVX4ISNS73MXSXIQDMGE4YB7W7YNM5NV3RFRMYQVLAA, seg_pos={0 0}, upload=557417]: Uploading piece... Jan 1 03:37:41.283101575 (636h8m41.15418461s) [noise=true , address=77.90.31.16:4735 , piece_id=AZGPEWUICPQSCS7WFXXKSW3ALI5IRKCMJDUSAWRK6LPWUESTIU4A, seg_pos={0 0}, upload=557417]: Uploading piece... Jan 1 03:37:41.283205664 (636h8m41.154288699s) [noise=true , address=145.79.227.2:4113 , piece_id=WCOX5PFXBPSNAUDO5OSOBNSPIV3UX5GUAJ4ORRSPQYP5RCAFJEZQ, seg_pos={0 0}, upload=557417]: Uploading piece... Jan 1 03:37:41.283479896 (636h8m41.154562941s) [noise=true , address=77.90.31.11:4554 , piece_id=744L33MQ3O5M222DKOK4Z6ZHRSRLH72FXFUACXVWTUKWQ6ZZGQIA, seg_pos={0 0}, upload=557417]: Uploading piece... Jan 1 03:37:41.283551833 (636h8m41.154634868s) [noise=true , address=145.79.227.6:4258 , piece_id=IG6IF6UJ6RLKKSTMJ6IISJNZHFZWHFX7GHXOXP44AAKZGV4GNSWQ, seg_pos={0 0}, upload=557417]: Uploading piece... Jan 1 03:37:41.283796208 (636h8m41.154879253s) [noise=true , address=45.91.116.5:4297 , piece_id=35SRYKGA36EQK6YQWINWYOAZPHZTHRPXWLUPR36SR2NNHW6CBITQ, seg_pos={0 0}, upload=557417]: Uploading piece... Jan 1 03:37:41.283884186 (636h8m41.154967221s) [noise=true , address=145.79.227.45:4230 , piece_id=2C3CMTNXCR5G2ZHV4REX4EGVEF4RE2OD5E2SEAPOZKBIG5OTXFLQ, seg_pos={0 0}, upload=557417]: Uploading piece... Jan 1 03:37:41.284141165 (636h8m41.1552242s) [noise=true , address=145.79.227.27:4689 , piece_id=MXHNHYBRABN23H43MCGSPV66VZO377NO5F7RYCSXL7HSGIAKYRXA, seg_pos={0 0}, upload=557417]: Uploading piece... Jan 1 03:37:41.284223702 (636h8m41.155306737s) [noise=true , address=145.79.227.42:4122 , piece_id=P24QWODKVFQPUKYCITZ5FEUDLRKIWS7OIW46UR2BXKS4Z2YDHB4Q, seg_pos={0 0}, upload=557417]: Uploading piece... Jan 1 03:37:41.284521690 (636h8m41.155604735s) [noise=true , address=77.90.31.13:4627 , piece_id=75QZDXBRQOV7SDDC5F6HTPUVM6EDV5MMTNROBFZNCNIR7QH3GVRA, seg_pos={0 0}, upload=557417]: Uploading piece... Jan 1 03:37:41.284696733 (636h8m41.155779778s) [noise=true , address=45.91.116.10:4082 , piece_id=RALN6RUHCQJ5H674VQD5J4NEJWKP2QFWRQBPNBB5OEORWD4YWUTA, seg_pos={0 0}, upload=557417]: Uploading piece... Jan 1 03:37:42.984632610 (636h8m42.855715645s) [upload=557417]: Next returned nil segment Jan 1 03:37:42.984647118 (636h8m42.855730153s) [upload=557417]: Waiting for error group managing segments... Jan 1 03:37:43.256857678 (636h8m43.127940723s) [noise=true , address=145.79.227.45:4230 , piece_id=2C3CMTNXCR5G2ZHV4REX4EGVEF4RE2OD5E2SEAPOZKBIG5OTXFLQ, seg_pos={0 0}, upload=557417]: Done uploading piece. err: <nil> Jan 1 03:37:43.266329344 (636h8m43.137412379s) [noise=true , address=145.79.227.41:4087 , piece_id=7SV7CLVJGNZQJNQJPXN2NBDPOQ24QYN654PX6MFTJ62IVGI27BOQ, seg_pos={0 0}, upload=557417]: Done uploading piece. err: <nil> Jan 1 03:37:43.269930430 (636h8m43.141013476s) [noise=true , address=145.79.227.2:4113 , piece_id=WCOX5PFXBPSNAUDO5OSOBNSPIV3UX5GUAJ4ORRSPQYP5RCAFJEZQ, seg_pos={0 0}, upload=557417]: Done uploading piece. err: <nil> Jan 1 03:37:43.275533440 (636h8m43.146616475s) [noise=true , address=145.79.227.27:4689 , piece_id=MXHNHYBRABN23H43MCGSPV66VZO377NO5F7RYCSXL7HSGIAKYRXA, seg_pos={0 0}, upload=557417]: Done uploading piece. err: <nil> Jan 1 03:37:43.282519421 (636h8m43.153602457s) [noise=true , address=145.79.227.6:4258 , piece_id=IG6IF6UJ6RLKKSTMJ6IISJNZHFZWHFX7GHXOXP44AAKZGV4GNSWQ, seg_pos={0 0}, upload=557417]: Done uploading piece. err: <nil> Jan 1 03:37:43.286136619 (636h8m43.157219654s) [noise=true , address=145.79.227.8:4331 , piece_id=JJVBSDXEGXOQ7N3GLYGU4BPRFS2FWLD7DHHPZ5EXNIUF2ATYSJKQ, seg_pos={0 0}, upload=557417]: Done uploading piece. err: <nil> Jan 1 03:37:43.289643076 (636h8m43.160726112s) [noise=true , address=145.79.227.31:4797 , piece_id=QEXJ5KUPNJZ5PAISMBNMPCHQL2ZQL5IEMRESHOIBOYMWG22NGVLQ, seg_pos={0 0}, upload=557417]: Done uploading piece. err: <nil> Jan 1 03:37:43.355713682 (636h8m43.226796727s) [noise=true , address=145.79.227.39:4013 , piece_id=LYFILPOG4UVX4ISNS73MXSXIQDMGE4YB7W7YNM5NV3RFRMYQVLAA, seg_pos={0 0}, upload=557417]: Done uploading piece. err: <nil> Jan 1 03:37:43.360679467 (636h8m43.231762512s) [noise=true , address=145.79.227.42:4122 , piece_id=P24QWODKVFQPUKYCITZ5FEUDLRKIWS7OIW46UR2BXKS4Z2YDHB4Q, seg_pos={0 0}, upload=557417]: Done uploading piece. err: <nil> Jan 1 03:37:43.547870661 (636h8m43.418953696s) [noise=true , address=77.90.31.11:4554 , piece_id=744L33MQ3O5M222DKOK4Z6ZHRSRLH72FXFUACXVWTUKWQ6ZZGQIA, seg_pos={0 0}, upload=557417]: Done uploading piece. err: <nil> Jan 1 03:37:43.584330184 (636h8m43.455413219s) [noise=true , address=77.90.31.18:4805 , piece_id=YHNGUE42NSLAIOMYWQ4SJ2C7O4BUKUAEXTYEIJZYCKKZCBMOSFQQ, seg_pos={0 0}, upload=557417]: Done uploading piece. err: <nil> Jan 1 03:37:43.615042174 (636h8m43.486125219s) [noise=true , address=77.90.31.16:4735 , piece_id=AZGPEWUICPQSCS7WFXXKSW3ALI5IRKCMJDUSAWRK6LPWUESTIU4A, seg_pos={0 0}, upload=557417]: Done uploading piece. err: <nil> Jan 1 03:37:43.687030308 (636h8m43.558113343s) [noise=true , address=45.91.116.5:4297 , piece_id=35SRYKGA36EQK6YQWINWYOAZPHZTHRPXWLUPR36SR2NNHW6CBITQ, seg_pos={0 0}, upload=557417]: Done uploading piece. err: <nil> Jan 1 03:37:43.687054985 (636h8m43.55813802s) [seg_pos={0 0}, upload=557417]: Segment reached optimal threshold of 13 pieces. Jan 1 03:37:43.687230439 (636h8m43.558313474s) [noise=true , address=77.90.31.13:4627 , piece_id=75QZDXBRQOV7SDDC5F6HTPUVM6EDV5MMTNROBFZNCNIR7QH3GVRA, seg_pos={0 0}, upload=557417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:55016->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:55016->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55016->77.90.31.13:4627: use of closed network connection Jan 1 03:37:43.687236180 (636h8m43.558319225s) [noise=true , address=77.90.31.13:4627 , piece_id=75QZDXBRQOV7SDDC5F6HTPUVM6EDV5MMTNROBFZNCNIR7QH3GVRA, seg_pos={0 0}, upload=557417]: Long tail cancellation due to optimal threshold Jan 1 03:37:43.687355567 (636h8m43.558438602s) [noise=true , address=45.91.116.4:4940 , piece_id=DVLVAHFQIIIJLUV5AC6X6CNPULNVLFQBTJ5BNUP2SMJV5Y2MSZJA, seg_pos={0 0}, upload=557417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:56256->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:56256->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56256->45.91.116.4:4940: use of closed network connection Jan 1 03:37:43.687362780 (636h8m43.558445815s) [noise=true , address=45.91.116.4:4940 , piece_id=DVLVAHFQIIIJLUV5AC6X6CNPULNVLFQBTJ5BNUP2SMJV5Y2MSZJA, seg_pos={0 0}, upload=557417]: Long tail cancellation due to optimal threshold Jan 1 03:37:43.687481136 (636h8m43.558564181s) [noise=true , address=45.91.116.10:4082 , piece_id=RALN6RUHCQJ5H674VQD5J4NEJWKP2QFWRQBPNBB5OEORWD4YWUTA, seg_pos={0 0}, upload=557417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jan 1 03:37:43.687487148 (636h8m43.558570193s) [noise=true , address=45.91.116.10:4082 , piece_id=RALN6RUHCQJ5H674VQD5J4NEJWKP2QFWRQBPNBB5OEORWD4YWUTA, seg_pos={0 0}, upload=557417]: Long tail cancellation due to optimal threshold Jan 1 03:37:43.687502867 (636h8m43.558585912s) [seg_pos={0 0}, upload=557417]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:37:43.687568442 (636h8m43.558651477s) [upload=557417]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:37:43.687573622 (636h8m43.558656667s) [upload=557417]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:37:43.980870394 (636h8m43.851953439s) [upload=557417]: Done uploading segments... Jan 1 03:37:43.980893307 (636h8m43.851976352s) [upload=557417]: Upload finished. err: <nil> Duration: 30.00314372s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X....X.......................................] duration: 42.122057693s trace: https://jaeger.dev.storj.io/trace/725fc16e891a4127 FAIL (2026-01-01 03:37:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3:03:23.412681369 (635h34m23.283764404s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413214023 (635h34m23.284297058s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413504245 (635h34m23.28458728s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413821750 (635h34m23.284904785s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414124316 (635h34m23.285207351s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414417554 (635h34m23.285500589s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414689172 (635h34m23.285772207s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415023338 (635h34m23.286106373s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415341384 (635h34m23.286424419s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415659309 (635h34m23.286742354s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416184289 (635h34m23.287267324s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416571025 (635h34m23.28765406s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416998890 (635h34m23.288081925s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.417376770 (635h34m23.288459815s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.418791664 (635h34m23.289874709s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.420183384 (635h34m23.291266419s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.421563200 (635h34m23.292646246s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:24.823751878 (635h34m24.694834913s) [upload=556915]: Next returned nil segment Jan 1 03:03:24.823765594 (635h34m24.694848629s) [upload=556915]: Waiting for error group managing segments... Jan 1 03:03:25.191603537 (635h34m25.062686582s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.271741648 (635h34m25.142824683s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.278781112 (635h34m25.149864157s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.337014705 (635h34m25.20809774s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.372659678 (635h34m25.243742713s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.439670894 (635h34m25.31075394s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.447380494 (635h34m25.318463529s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.459093186 (635h34m25.330176231s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.602592395 (635h34m25.47367543s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.719366504 (635h34m25.59044954s) [noise=true , address=45.91.116.12:4230 , piece_id=WQHO3KOUTZSHBZIDUXANBVFSUKTBKEWHTZ3Z7BNGXGOFP2BMJHWA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.875237800 (635h34m25.746320845s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.143929177 (635h34m26.015012212s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.159675169 (635h34m26.030758204s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181352455 (635h34m26.05243549s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181379146 (635h34m26.052462181s) [seg_pos={0 0}, upload=556915]: Segment reached optimal threshold of 14 pieces. Jan 1 03:03:26.181716378 (635h34m26.052799413s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Pvm1SftEF3tnCEBqUn1ZmyVcLDBkR6pyegEb7mtzjdQiiqYqt): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181723281 (635h34m26.052806316s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181784638 (635h34m26.052867703s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181848530 (635h34m26.052931565s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181859921 (635h34m26.052942956s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181866474 (635h34m26.052949509s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181920486 (635h34m26.053003522s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:184g6qHdezLW3d2CKNW8GRgdzeVLQuMEhd8B8c1zEnkKNkeSTu): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181925296 (635h34m26.053008331s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181941377 (635h34m26.053024412s) [seg_pos={0 0}, upload=556915]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 03:03:26.182003695 (635h34m26.05308673s) [upload=556915]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:03:26.182009416 (635h34m26.053092461s) [upload=556915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:03:26.474026257 (635h34m26.345109292s) [upload=556915]: Done uploading segments... Jan 1 03:03:26.474049391 (635h34m26.345132427s) [upload=556915]: Upload finished. err: <nil> Duration: 42.122092498s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.................X......X............................X......] duration: 3.486978585s trace: https://jaeger.dev.storj.io/trace/2ad779a295dfac2 FAIL (2026-01-01 03:37:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.487005527s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X....X......................................] duration: 45.405673778s trace: https://jaeger.dev.storj.io/trace/725fc16e891a4127 FAIL (2026-01-01 03:37:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3:03:23.412681369 (635h34m23.283764404s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413214023 (635h34m23.284297058s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413504245 (635h34m23.28458728s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413821750 (635h34m23.284904785s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414124316 (635h34m23.285207351s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414417554 (635h34m23.285500589s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414689172 (635h34m23.285772207s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415023338 (635h34m23.286106373s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415341384 (635h34m23.286424419s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415659309 (635h34m23.286742354s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416184289 (635h34m23.287267324s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416571025 (635h34m23.28765406s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416998890 (635h34m23.288081925s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.417376770 (635h34m23.288459815s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.418791664 (635h34m23.289874709s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.420183384 (635h34m23.291266419s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.421563200 (635h34m23.292646246s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:24.823751878 (635h34m24.694834913s) [upload=556915]: Next returned nil segment Jan 1 03:03:24.823765594 (635h34m24.694848629s) [upload=556915]: Waiting for error group managing segments... Jan 1 03:03:25.191603537 (635h34m25.062686582s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.271741648 (635h34m25.142824683s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.278781112 (635h34m25.149864157s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.337014705 (635h34m25.20809774s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.372659678 (635h34m25.243742713s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.439670894 (635h34m25.31075394s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.447380494 (635h34m25.318463529s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.459093186 (635h34m25.330176231s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.602592395 (635h34m25.47367543s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.719366504 (635h34m25.59044954s) [noise=true , address=45.91.116.12:4230 , piece_id=WQHO3KOUTZSHBZIDUXANBVFSUKTBKEWHTZ3Z7BNGXGOFP2BMJHWA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.875237800 (635h34m25.746320845s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.143929177 (635h34m26.015012212s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.159675169 (635h34m26.030758204s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181352455 (635h34m26.05243549s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181379146 (635h34m26.052462181s) [seg_pos={0 0}, upload=556915]: Segment reached optimal threshold of 14 pieces. Jan 1 03:03:26.181716378 (635h34m26.052799413s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Pvm1SftEF3tnCEBqUn1ZmyVcLDBkR6pyegEb7mtzjdQiiqYqt): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181723281 (635h34m26.052806316s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181784638 (635h34m26.052867703s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181848530 (635h34m26.052931565s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181859921 (635h34m26.052942956s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181866474 (635h34m26.052949509s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181920486 (635h34m26.053003522s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:184g6qHdezLW3d2CKNW8GRgdzeVLQuMEhd8B8c1zEnkKNkeSTu): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181925296 (635h34m26.053008331s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181941377 (635h34m26.053024412s) [seg_pos={0 0}, upload=556915]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 03:03:26.182003695 (635h34m26.05308673s) [upload=556915]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:03:26.182009416 (635h34m26.053092461s) [upload=556915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:03:26.474026257 (635h34m26.345109292s) [upload=556915]: Done uploading segments... Jan 1 03:03:26.474049391 (635h34m26.345132427s) [upload=556915]: Upload finished. err: <nil> Duration: 45.405704056s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [......X...X..X..X...X..X....XXX.X.X..XX.X.XXX..XX..XXXXX....] duration: 7.244847032s trace: https://jaeger.dev.storj.io/trace/511c0acf1687258e FAIL (2026-01-01 03:37:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: YJVA6TFSANAM4GXNJL57DECMD5WWPI6H4L3PAUMLMEJANGGREGUA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: MOMILYJ6A6EMTVHHO27LNNPQECIQYQ7L4OJN5OX5PKVTH2NCE3GA) context deadline exceeded ======LOGS====== Jan 1 03:33:56.314733719 (636h4m56.185816764s) [upload=557361]: Starting upload Jan 1 03:33:56.314856132 (636h4m56.185939177s) [upload=557361]: Done starting upload Jan 1 03:33:56.314936174 (636h4m56.186019209s) [upload=557361]: Uploading segments... Jan 1 03:33:56.315127339 (636h4m56.186210374s) [upload=557361]: Got next segment. Inline: false Jan 1 03:33:56.315162365 (636h4m56.186245411s) [upload=557361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:33:56.315167675 (636h4m56.186250721s) [upload=557361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:33:56.585484360 (636h4m56.456567405s) [seg_pos={0 0}, upload=557361]: Begin upload segment... Jan 1 03:33:56.585563410 (636h4m56.456646455s) [seg_pos={0 0}, upload=557361]: Done begin upload segment. Jan 1 03:33:56.585600531 (636h4m56.456683576s) [noise=true , address=145.79.227.6:4257 , piece_id=X3UE3Z55HMVVFMHXITIC5SAJGIH2J5QFDYVNY66CP2VS7DFHYMIA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.585748682 (636h4m56.456831728s) [noise=true , address=45.91.116.5:4300 , piece_id=HJKG5RL5KWVCBVP7VJ6IDAIRDSKTHED6FOIAXIOL5FF32XLKRCEQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.586623668 (636h4m56.457706704s) [noise=true , address=145.79.227.39:4013 , piece_id=NQZ7PMFQRYM2WWBSFXJBP26FMQZY6IAWCXHMX5J4GYYKPOLBHHJQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.587286000 (636h4m56.458369035s) [noise=true , address=45.91.116.13:4156 , piece_id=MOMILYJ6A6EMTVHHO27LNNPQECIQYQ7L4OJN5OX5PKVTH2NCE3GA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.587953671 (636h4m56.459036716s) [noise=true , address=45.91.116.14:4192 , piece_id=YJVA6TFSANAM4GXNJL57DECMD5WWPI6H4L3PAUMLMEJANGGREGUA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.588896376 (636h4m56.459979421s) [noise=true , address=145.79.227.41:4085 , piece_id=234273FC25CEB4BGKAHDL6MMQAO6H7LNTYBTJMKAHHB35NHUWJTA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.590458651 (636h4m56.461541716s) [noise=true , address=145.79.227.12:4767 , piece_id=Q5MKSZLVNH4AOSJ4O7PJ27L7OWCF2XNME4UXLRQ6IGCWNN5CILHA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.592544222 (636h4m56.463627297s) [noise=true , address=145.79.227.15:4873 , piece_id=7BEE44HJU6KPVZW5UII3JG6SPCTCKRK4BSLSM7UANTDYMBM4VD5Q, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.593386686 (636h4m56.464469731s) [noise=true , address=145.79.227.5:4222 , piece_id=D3XD4KVOFPOYWMRGC5UVDG3JE3LLS4TKKR6BXL6HEF2DJPGO6JSQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.594256092 (636h4m56.465339127s) [noise=true , address=45.91.116.10:4081 , piece_id=NNABYWDFRZEJENTE7UPYV4UEY55E7MWFDAHQZP4TCYPYMHQ7ODBQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.594836167 (636h4m56.465919202s) [noise=true , address=145.79.227.10:4403 , piece_id=M3K5HHL7R6JRQMFUW4TCQDOZOWRGKLNYBVYQA2YD6AM37DW2L5HA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.595705282 (636h4m56.466788327s) [noise=true , address=145.79.227.11:4729 , piece_id=PL6XQ6DNCVGLSXLCCAWYY4OCSALV3KXMADICWEVB3I2CSMHQICZA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.596215913 (636h4m56.467298948s) [noise=true , address=45.91.116.11:4120 , piece_id=CYPIFYENY23MG4BOVP7QV65EHGTW354LRKO4VRCXJ6C5G6IU2RAA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.596517317 (636h4m56.467600363s) [noise=true , address=45.91.116.12:4226 , piece_id=G2LGRPHRD7HA6HJHS7IMYU4XNMOSHHNXYN3GJBA6CTT3IE6ET7LQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.598204309 (636h4m56.469287344s) [noise=true , address=145.79.227.8:4330 , piece_id=AAGIVTKRUXB3ZEW5UGG6VXSRCXP5ZOO67BNPUT7OTPEB7E63TZTQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.600691746 (636h4m56.471774791s) [noise=true , address=77.90.31.10:4520 , piece_id=PYPXCHEPUFQYDOWDOZ5WDDN374KOLMTPBBQRBPHTUEYHW374UL3A, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:58.281710480 (636h4m58.152793525s) [upload=557361]: Next returned nil segment Jan 1 03:33:58.281724658 (636h4m58.152807693s) [upload=557361]: Waiting for error group managing segments... Jan 1 03:33:58.566383030 (636h4m58.437466075s) [noise=true , address=145.79.227.41:4085 , piece_id=234273FC25CEB4BGKAHDL6MMQAO6H7LNTYBTJMKAHHB35NHUWJTA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.567784098 (636h4m58.438867173s) [noise=true , address=145.79.227.15:4873 , piece_id=7BEE44HJU6KPVZW5UII3JG6SPCTCKRK4BSLSM7UANTDYMBM4VD5Q, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.583853285 (636h4m58.45493633s) [noise=true , address=145.79.227.12:4767 , piece_id=Q5MKSZLVNH4AOSJ4O7PJ27L7OWCF2XNME4UXLRQ6IGCWNN5CILHA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.591537447 (636h4m58.462620492s) [noise=true , address=145.79.227.39:4013 , piece_id=NQZ7PMFQRYM2WWBSFXJBP26FMQZY6IAWCXHMX5J4GYYKPOLBHHJQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.592304357 (636h4m58.463387392s) [noise=true , address=145.79.227.6:4257 , piece_id=X3UE3Z55HMVVFMHXITIC5SAJGIH2J5QFDYVNY66CP2VS7DFHYMIA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.599833965 (636h4m58.47091701s) [noise=true , address=145.79.227.11:4729 , piece_id=PL6XQ6DNCVGLSXLCCAWYY4OCSALV3KXMADICWEVB3I2CSMHQICZA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.601388594 (636h4m58.472471639s) [noise=true , address=145.79.227.10:4403 , piece_id=M3K5HHL7R6JRQMFUW4TCQDOZOWRGKLNYBVYQA2YD6AM37DW2L5HA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.630591339 (636h4m58.501674374s) [noise=true , address=145.79.227.8:4330 , piece_id=AAGIVTKRUXB3ZEW5UGG6VXSRCXP5ZOO67BNPUT7OTPEB7E63TZTQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.650577436 (636h4m58.521660481s) [noise=true , address=145.79.227.5:4222 , piece_id=D3XD4KVOFPOYWMRGC5UVDG3JE3LLS4TKKR6BXL6HEF2DJPGO6JSQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.916297107 (636h4m58.787380152s) [noise=true , address=45.91.116.12:4226 , piece_id=G2LGRPHRD7HA6HJHS7IMYU4XNMOSHHNXYN3GJBA6CTT3IE6ET7LQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.920798298 (636h4m58.791881333s) [noise=true , address=45.91.116.13:4156 , piece_id=MOMILYJ6A6EMTVHHO27LNNPQECIQYQ7L4OJN5OX5PKVTH2NCE3GA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.921432575 (636h4m58.792515621s) [noise=true , address=45.91.116.14:4192 , piece_id=YJVA6TFSANAM4GXNJL57DECMD5WWPI6H4L3PAUMLMEJANGGREGUA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.955733448 (636h4m58.826816493s) [noise=true , address=45.91.116.10:4081 , piece_id=NNABYWDFRZEJENTE7UPYV4UEY55E7MWFDAHQZP4TCYPYMHQ7ODBQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.955763956 (636h4m58.826846991s) [seg_pos={0 0}, upload=557361]: Segment reached optimal threshold of 13 pieces. Jan 1 03:33:58.956096580 (636h4m58.827179625s) [noise=true , address=45.91.116.11:4120 , piece_id=CYPIFYENY23MG4BOVP7QV65EHGTW354LRKO4VRCXJ6C5G6IU2RAA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Jan 1 03:33:58.956105156 (636h4m58.827188201s) [noise=true , address=45.91.116.11:4120 , piece_id=CYPIFYENY23MG4BOVP7QV65EHGTW354LRKO4VRCXJ6C5G6IU2RAA, seg_pos={0 0}, upload=557361]: Long tail cancellation due to optimal threshold Jan 1 03:33:58.956367546 (636h4m58.827450591s) [noise=true , address=77.90.31.10:4520 , piece_id=PYPXCHEPUFQYDOWDOZ5WDDN374KOLMTPBBQRBPHTUEYHW374UL3A, seg_pos={0 0}, upload=557361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Jan 1 03:33:58.956391661 (636h4m58.827474706s) [noise=true , address=77.90.31.10:4520 , piece_id=PYPXCHEPUFQYDOWDOZ5WDDN374KOLMTPBBQRBPHTUEYHW374UL3A, seg_pos={0 0}, upload=557361]: Long tail cancellation due to optimal threshold Jan 1 03:33:58.956461695 (636h4m58.82754476s) [noise=true , address=45.91.116.5:4300 , piece_id=HJKG5RL5KWVCBVP7VJ6IDAIRDSKTHED6FOIAXIOL5FF32XLKRCEQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:37472->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:37472->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37472->45.91.116.5:4300: use of closed network connection Jan 1 03:33:58.956474369 (636h4m58.827557404s) [noise=true , address=45.91.116.5:4300 , piece_id=HJKG5RL5KWVCBVP7VJ6IDAIRDSKTHED6FOIAXIOL5FF32XLKRCEQ, seg_pos={0 0}, upload=557361]: Long tail cancellation due to optimal threshold Jan 1 03:33:58.956524344 (636h4m58.827607389s) [seg_pos={0 0}, upload=557361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:33:58.956658219 (636h4m58.827741274s) [upload=557361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:33:58.956667076 (636h4m58.827750111s) [upload=557361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:33:59.288606697 (636h4m59.159689732s) [upload=557361]: Done uploading segments... Jan 1 03:33:59.288632837 (636h4m59.159715882s) [upload=557361]: Upload finished. err: <nil> Duration: 7.244885054s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..................X......X............................X.....] duration: 3.542189467s trace: https://jaeger.dev.storj.io/trace/2ad779a295dfac2 FAIL (2026-01-01 03:36:59 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.542220525s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X....X.....................................] duration: 36.589320083s trace: https://jaeger.dev.storj.io/trace/725fc16e891a4127 FAIL (2026-01-01 03:36:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3:03:23.412681369 (635h34m23.283764404s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413214023 (635h34m23.284297058s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413504245 (635h34m23.28458728s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413821750 (635h34m23.284904785s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414124316 (635h34m23.285207351s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414417554 (635h34m23.285500589s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414689172 (635h34m23.285772207s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415023338 (635h34m23.286106373s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415341384 (635h34m23.286424419s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415659309 (635h34m23.286742354s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416184289 (635h34m23.287267324s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416571025 (635h34m23.28765406s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416998890 (635h34m23.288081925s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.417376770 (635h34m23.288459815s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.418791664 (635h34m23.289874709s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.420183384 (635h34m23.291266419s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.421563200 (635h34m23.292646246s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:24.823751878 (635h34m24.694834913s) [upload=556915]: Next returned nil segment Jan 1 03:03:24.823765594 (635h34m24.694848629s) [upload=556915]: Waiting for error group managing segments... Jan 1 03:03:25.191603537 (635h34m25.062686582s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.271741648 (635h34m25.142824683s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.278781112 (635h34m25.149864157s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.337014705 (635h34m25.20809774s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.372659678 (635h34m25.243742713s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.439670894 (635h34m25.31075394s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.447380494 (635h34m25.318463529s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.459093186 (635h34m25.330176231s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.602592395 (635h34m25.47367543s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.719366504 (635h34m25.59044954s) [noise=true , address=45.91.116.12:4230 , piece_id=WQHO3KOUTZSHBZIDUXANBVFSUKTBKEWHTZ3Z7BNGXGOFP2BMJHWA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.875237800 (635h34m25.746320845s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.143929177 (635h34m26.015012212s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.159675169 (635h34m26.030758204s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181352455 (635h34m26.05243549s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181379146 (635h34m26.052462181s) [seg_pos={0 0}, upload=556915]: Segment reached optimal threshold of 14 pieces. Jan 1 03:03:26.181716378 (635h34m26.052799413s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Pvm1SftEF3tnCEBqUn1ZmyVcLDBkR6pyegEb7mtzjdQiiqYqt): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181723281 (635h34m26.052806316s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181784638 (635h34m26.052867703s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181848530 (635h34m26.052931565s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181859921 (635h34m26.052942956s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181866474 (635h34m26.052949509s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181920486 (635h34m26.053003522s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:184g6qHdezLW3d2CKNW8GRgdzeVLQuMEhd8B8c1zEnkKNkeSTu): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181925296 (635h34m26.053008331s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181941377 (635h34m26.053024412s) [seg_pos={0 0}, upload=556915]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 03:03:26.182003695 (635h34m26.05308673s) [upload=556915]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:03:26.182009416 (635h34m26.053092461s) [upload=556915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:03:26.474026257 (635h34m26.345109292s) [upload=556915]: Done uploading segments... Jan 1 03:03:26.474049391 (635h34m26.345132427s) [upload=556915]: Upload finished. err: <nil> Duration: 36.589349659s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.......X...X..X..X...X..X....XXX.X.X..XX.X.XXX..XX..XXXXX...] duration: 5.991020187s trace: https://jaeger.dev.storj.io/trace/511c0acf1687258e FAIL (2026-01-01 03:36:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: YJVA6TFSANAM4GXNJL57DECMD5WWPI6H4L3PAUMLMEJANGGREGUA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: MOMILYJ6A6EMTVHHO27LNNPQECIQYQ7L4OJN5OX5PKVTH2NCE3GA) context deadline exceeded ======LOGS====== Jan 1 03:33:56.314733719 (636h4m56.185816764s) [upload=557361]: Starting upload Jan 1 03:33:56.314856132 (636h4m56.185939177s) [upload=557361]: Done starting upload Jan 1 03:33:56.314936174 (636h4m56.186019209s) [upload=557361]: Uploading segments... Jan 1 03:33:56.315127339 (636h4m56.186210374s) [upload=557361]: Got next segment. Inline: false Jan 1 03:33:56.315162365 (636h4m56.186245411s) [upload=557361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:33:56.315167675 (636h4m56.186250721s) [upload=557361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:33:56.585484360 (636h4m56.456567405s) [seg_pos={0 0}, upload=557361]: Begin upload segment... Jan 1 03:33:56.585563410 (636h4m56.456646455s) [seg_pos={0 0}, upload=557361]: Done begin upload segment. Jan 1 03:33:56.585600531 (636h4m56.456683576s) [noise=true , address=145.79.227.6:4257 , piece_id=X3UE3Z55HMVVFMHXITIC5SAJGIH2J5QFDYVNY66CP2VS7DFHYMIA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.585748682 (636h4m56.456831728s) [noise=true , address=45.91.116.5:4300 , piece_id=HJKG5RL5KWVCBVP7VJ6IDAIRDSKTHED6FOIAXIOL5FF32XLKRCEQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.586623668 (636h4m56.457706704s) [noise=true , address=145.79.227.39:4013 , piece_id=NQZ7PMFQRYM2WWBSFXJBP26FMQZY6IAWCXHMX5J4GYYKPOLBHHJQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.587286000 (636h4m56.458369035s) [noise=true , address=45.91.116.13:4156 , piece_id=MOMILYJ6A6EMTVHHO27LNNPQECIQYQ7L4OJN5OX5PKVTH2NCE3GA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.587953671 (636h4m56.459036716s) [noise=true , address=45.91.116.14:4192 , piece_id=YJVA6TFSANAM4GXNJL57DECMD5WWPI6H4L3PAUMLMEJANGGREGUA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.588896376 (636h4m56.459979421s) [noise=true , address=145.79.227.41:4085 , piece_id=234273FC25CEB4BGKAHDL6MMQAO6H7LNTYBTJMKAHHB35NHUWJTA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.590458651 (636h4m56.461541716s) [noise=true , address=145.79.227.12:4767 , piece_id=Q5MKSZLVNH4AOSJ4O7PJ27L7OWCF2XNME4UXLRQ6IGCWNN5CILHA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.592544222 (636h4m56.463627297s) [noise=true , address=145.79.227.15:4873 , piece_id=7BEE44HJU6KPVZW5UII3JG6SPCTCKRK4BSLSM7UANTDYMBM4VD5Q, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.593386686 (636h4m56.464469731s) [noise=true , address=145.79.227.5:4222 , piece_id=D3XD4KVOFPOYWMRGC5UVDG3JE3LLS4TKKR6BXL6HEF2DJPGO6JSQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.594256092 (636h4m56.465339127s) [noise=true , address=45.91.116.10:4081 , piece_id=NNABYWDFRZEJENTE7UPYV4UEY55E7MWFDAHQZP4TCYPYMHQ7ODBQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.594836167 (636h4m56.465919202s) [noise=true , address=145.79.227.10:4403 , piece_id=M3K5HHL7R6JRQMFUW4TCQDOZOWRGKLNYBVYQA2YD6AM37DW2L5HA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.595705282 (636h4m56.466788327s) [noise=true , address=145.79.227.11:4729 , piece_id=PL6XQ6DNCVGLSXLCCAWYY4OCSALV3KXMADICWEVB3I2CSMHQICZA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.596215913 (636h4m56.467298948s) [noise=true , address=45.91.116.11:4120 , piece_id=CYPIFYENY23MG4BOVP7QV65EHGTW354LRKO4VRCXJ6C5G6IU2RAA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.596517317 (636h4m56.467600363s) [noise=true , address=45.91.116.12:4226 , piece_id=G2LGRPHRD7HA6HJHS7IMYU4XNMOSHHNXYN3GJBA6CTT3IE6ET7LQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.598204309 (636h4m56.469287344s) [noise=true , address=145.79.227.8:4330 , piece_id=AAGIVTKRUXB3ZEW5UGG6VXSRCXP5ZOO67BNPUT7OTPEB7E63TZTQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.600691746 (636h4m56.471774791s) [noise=true , address=77.90.31.10:4520 , piece_id=PYPXCHEPUFQYDOWDOZ5WDDN374KOLMTPBBQRBPHTUEYHW374UL3A, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:58.281710480 (636h4m58.152793525s) [upload=557361]: Next returned nil segment Jan 1 03:33:58.281724658 (636h4m58.152807693s) [upload=557361]: Waiting for error group managing segments... Jan 1 03:33:58.566383030 (636h4m58.437466075s) [noise=true , address=145.79.227.41:4085 , piece_id=234273FC25CEB4BGKAHDL6MMQAO6H7LNTYBTJMKAHHB35NHUWJTA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.567784098 (636h4m58.438867173s) [noise=true , address=145.79.227.15:4873 , piece_id=7BEE44HJU6KPVZW5UII3JG6SPCTCKRK4BSLSM7UANTDYMBM4VD5Q, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.583853285 (636h4m58.45493633s) [noise=true , address=145.79.227.12:4767 , piece_id=Q5MKSZLVNH4AOSJ4O7PJ27L7OWCF2XNME4UXLRQ6IGCWNN5CILHA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.591537447 (636h4m58.462620492s) [noise=true , address=145.79.227.39:4013 , piece_id=NQZ7PMFQRYM2WWBSFXJBP26FMQZY6IAWCXHMX5J4GYYKPOLBHHJQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.592304357 (636h4m58.463387392s) [noise=true , address=145.79.227.6:4257 , piece_id=X3UE3Z55HMVVFMHXITIC5SAJGIH2J5QFDYVNY66CP2VS7DFHYMIA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.599833965 (636h4m58.47091701s) [noise=true , address=145.79.227.11:4729 , piece_id=PL6XQ6DNCVGLSXLCCAWYY4OCSALV3KXMADICWEVB3I2CSMHQICZA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.601388594 (636h4m58.472471639s) [noise=true , address=145.79.227.10:4403 , piece_id=M3K5HHL7R6JRQMFUW4TCQDOZOWRGKLNYBVYQA2YD6AM37DW2L5HA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.630591339 (636h4m58.501674374s) [noise=true , address=145.79.227.8:4330 , piece_id=AAGIVTKRUXB3ZEW5UGG6VXSRCXP5ZOO67BNPUT7OTPEB7E63TZTQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.650577436 (636h4m58.521660481s) [noise=true , address=145.79.227.5:4222 , piece_id=D3XD4KVOFPOYWMRGC5UVDG3JE3LLS4TKKR6BXL6HEF2DJPGO6JSQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.916297107 (636h4m58.787380152s) [noise=true , address=45.91.116.12:4226 , piece_id=G2LGRPHRD7HA6HJHS7IMYU4XNMOSHHNXYN3GJBA6CTT3IE6ET7LQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.920798298 (636h4m58.791881333s) [noise=true , address=45.91.116.13:4156 , piece_id=MOMILYJ6A6EMTVHHO27LNNPQECIQYQ7L4OJN5OX5PKVTH2NCE3GA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.921432575 (636h4m58.792515621s) [noise=true , address=45.91.116.14:4192 , piece_id=YJVA6TFSANAM4GXNJL57DECMD5WWPI6H4L3PAUMLMEJANGGREGUA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.955733448 (636h4m58.826816493s) [noise=true , address=45.91.116.10:4081 , piece_id=NNABYWDFRZEJENTE7UPYV4UEY55E7MWFDAHQZP4TCYPYMHQ7ODBQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.955763956 (636h4m58.826846991s) [seg_pos={0 0}, upload=557361]: Segment reached optimal threshold of 13 pieces. Jan 1 03:33:58.956096580 (636h4m58.827179625s) [noise=true , address=45.91.116.11:4120 , piece_id=CYPIFYENY23MG4BOVP7QV65EHGTW354LRKO4VRCXJ6C5G6IU2RAA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Jan 1 03:33:58.956105156 (636h4m58.827188201s) [noise=true , address=45.91.116.11:4120 , piece_id=CYPIFYENY23MG4BOVP7QV65EHGTW354LRKO4VRCXJ6C5G6IU2RAA, seg_pos={0 0}, upload=557361]: Long tail cancellation due to optimal threshold Jan 1 03:33:58.956367546 (636h4m58.827450591s) [noise=true , address=77.90.31.10:4520 , piece_id=PYPXCHEPUFQYDOWDOZ5WDDN374KOLMTPBBQRBPHTUEYHW374UL3A, seg_pos={0 0}, upload=557361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Jan 1 03:33:58.956391661 (636h4m58.827474706s) [noise=true , address=77.90.31.10:4520 , piece_id=PYPXCHEPUFQYDOWDOZ5WDDN374KOLMTPBBQRBPHTUEYHW374UL3A, seg_pos={0 0}, upload=557361]: Long tail cancellation due to optimal threshold Jan 1 03:33:58.956461695 (636h4m58.82754476s) [noise=true , address=45.91.116.5:4300 , piece_id=HJKG5RL5KWVCBVP7VJ6IDAIRDSKTHED6FOIAXIOL5FF32XLKRCEQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:37472->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:37472->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37472->45.91.116.5:4300: use of closed network connection Jan 1 03:33:58.956474369 (636h4m58.827557404s) [noise=true , address=45.91.116.5:4300 , piece_id=HJKG5RL5KWVCBVP7VJ6IDAIRDSKTHED6FOIAXIOL5FF32XLKRCEQ, seg_pos={0 0}, upload=557361]: Long tail cancellation due to optimal threshold Jan 1 03:33:58.956524344 (636h4m58.827607389s) [seg_pos={0 0}, upload=557361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:33:58.956658219 (636h4m58.827741274s) [upload=557361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:33:58.956667076 (636h4m58.827750111s) [upload=557361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:33:59.288606697 (636h4m59.159689732s) [upload=557361]: Done uploading segments... Jan 1 03:33:59.288632837 (636h4m59.159715882s) [upload=557361]: Upload finished. err: <nil> Duration: 5.991073608s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...................X......X............................X....] duration: 3.674382762s trace: https://jaeger.dev.storj.io/trace/2ad779a295dfac2 FAIL (2026-01-01 03:35:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.674408841s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X....X....................................] duration: 36.570470114s trace: https://jaeger.dev.storj.io/trace/725fc16e891a4127 FAIL (2026-01-01 03:35:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3:03:23.412681369 (635h34m23.283764404s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413214023 (635h34m23.284297058s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413504245 (635h34m23.28458728s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413821750 (635h34m23.284904785s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414124316 (635h34m23.285207351s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414417554 (635h34m23.285500589s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414689172 (635h34m23.285772207s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415023338 (635h34m23.286106373s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415341384 (635h34m23.286424419s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415659309 (635h34m23.286742354s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416184289 (635h34m23.287267324s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416571025 (635h34m23.28765406s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416998890 (635h34m23.288081925s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.417376770 (635h34m23.288459815s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.418791664 (635h34m23.289874709s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.420183384 (635h34m23.291266419s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.421563200 (635h34m23.292646246s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:24.823751878 (635h34m24.694834913s) [upload=556915]: Next returned nil segment Jan 1 03:03:24.823765594 (635h34m24.694848629s) [upload=556915]: Waiting for error group managing segments... Jan 1 03:03:25.191603537 (635h34m25.062686582s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.271741648 (635h34m25.142824683s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.278781112 (635h34m25.149864157s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.337014705 (635h34m25.20809774s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.372659678 (635h34m25.243742713s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.439670894 (635h34m25.31075394s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.447380494 (635h34m25.318463529s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.459093186 (635h34m25.330176231s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.602592395 (635h34m25.47367543s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.719366504 (635h34m25.59044954s) [noise=true , address=45.91.116.12:4230 , piece_id=WQHO3KOUTZSHBZIDUXANBVFSUKTBKEWHTZ3Z7BNGXGOFP2BMJHWA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.875237800 (635h34m25.746320845s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.143929177 (635h34m26.015012212s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.159675169 (635h34m26.030758204s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181352455 (635h34m26.05243549s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181379146 (635h34m26.052462181s) [seg_pos={0 0}, upload=556915]: Segment reached optimal threshold of 14 pieces. Jan 1 03:03:26.181716378 (635h34m26.052799413s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Pvm1SftEF3tnCEBqUn1ZmyVcLDBkR6pyegEb7mtzjdQiiqYqt): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181723281 (635h34m26.052806316s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181784638 (635h34m26.052867703s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181848530 (635h34m26.052931565s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181859921 (635h34m26.052942956s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181866474 (635h34m26.052949509s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181920486 (635h34m26.053003522s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:184g6qHdezLW3d2CKNW8GRgdzeVLQuMEhd8B8c1zEnkKNkeSTu): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181925296 (635h34m26.053008331s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181941377 (635h34m26.053024412s) [seg_pos={0 0}, upload=556915]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 03:03:26.182003695 (635h34m26.05308673s) [upload=556915]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:03:26.182009416 (635h34m26.053092461s) [upload=556915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:03:26.474026257 (635h34m26.345109292s) [upload=556915]: Done uploading segments... Jan 1 03:03:26.474049391 (635h34m26.345132427s) [upload=556915]: Upload finished. err: <nil> Duration: 36.570485604s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X.......X...X..X..X...X..X....XXX.X.X..XX.X.XXX..XX..XXXXX..] duration: 8.047913459s trace: https://jaeger.dev.storj.io/trace/511c0acf1687258e FAIL (2026-01-01 03:35:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: YJVA6TFSANAM4GXNJL57DECMD5WWPI6H4L3PAUMLMEJANGGREGUA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: MOMILYJ6A6EMTVHHO27LNNPQECIQYQ7L4OJN5OX5PKVTH2NCE3GA) context deadline exceeded ======LOGS====== Jan 1 03:33:56.314733719 (636h4m56.185816764s) [upload=557361]: Starting upload Jan 1 03:33:56.314856132 (636h4m56.185939177s) [upload=557361]: Done starting upload Jan 1 03:33:56.314936174 (636h4m56.186019209s) [upload=557361]: Uploading segments... Jan 1 03:33:56.315127339 (636h4m56.186210374s) [upload=557361]: Got next segment. Inline: false Jan 1 03:33:56.315162365 (636h4m56.186245411s) [upload=557361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:33:56.315167675 (636h4m56.186250721s) [upload=557361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:33:56.585484360 (636h4m56.456567405s) [seg_pos={0 0}, upload=557361]: Begin upload segment... Jan 1 03:33:56.585563410 (636h4m56.456646455s) [seg_pos={0 0}, upload=557361]: Done begin upload segment. Jan 1 03:33:56.585600531 (636h4m56.456683576s) [noise=true , address=145.79.227.6:4257 , piece_id=X3UE3Z55HMVVFMHXITIC5SAJGIH2J5QFDYVNY66CP2VS7DFHYMIA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.585748682 (636h4m56.456831728s) [noise=true , address=45.91.116.5:4300 , piece_id=HJKG5RL5KWVCBVP7VJ6IDAIRDSKTHED6FOIAXIOL5FF32XLKRCEQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.586623668 (636h4m56.457706704s) [noise=true , address=145.79.227.39:4013 , piece_id=NQZ7PMFQRYM2WWBSFXJBP26FMQZY6IAWCXHMX5J4GYYKPOLBHHJQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.587286000 (636h4m56.458369035s) [noise=true , address=45.91.116.13:4156 , piece_id=MOMILYJ6A6EMTVHHO27LNNPQECIQYQ7L4OJN5OX5PKVTH2NCE3GA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.587953671 (636h4m56.459036716s) [noise=true , address=45.91.116.14:4192 , piece_id=YJVA6TFSANAM4GXNJL57DECMD5WWPI6H4L3PAUMLMEJANGGREGUA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.588896376 (636h4m56.459979421s) [noise=true , address=145.79.227.41:4085 , piece_id=234273FC25CEB4BGKAHDL6MMQAO6H7LNTYBTJMKAHHB35NHUWJTA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.590458651 (636h4m56.461541716s) [noise=true , address=145.79.227.12:4767 , piece_id=Q5MKSZLVNH4AOSJ4O7PJ27L7OWCF2XNME4UXLRQ6IGCWNN5CILHA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.592544222 (636h4m56.463627297s) [noise=true , address=145.79.227.15:4873 , piece_id=7BEE44HJU6KPVZW5UII3JG6SPCTCKRK4BSLSM7UANTDYMBM4VD5Q, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.593386686 (636h4m56.464469731s) [noise=true , address=145.79.227.5:4222 , piece_id=D3XD4KVOFPOYWMRGC5UVDG3JE3LLS4TKKR6BXL6HEF2DJPGO6JSQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.594256092 (636h4m56.465339127s) [noise=true , address=45.91.116.10:4081 , piece_id=NNABYWDFRZEJENTE7UPYV4UEY55E7MWFDAHQZP4TCYPYMHQ7ODBQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.594836167 (636h4m56.465919202s) [noise=true , address=145.79.227.10:4403 , piece_id=M3K5HHL7R6JRQMFUW4TCQDOZOWRGKLNYBVYQA2YD6AM37DW2L5HA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.595705282 (636h4m56.466788327s) [noise=true , address=145.79.227.11:4729 , piece_id=PL6XQ6DNCVGLSXLCCAWYY4OCSALV3KXMADICWEVB3I2CSMHQICZA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.596215913 (636h4m56.467298948s) [noise=true , address=45.91.116.11:4120 , piece_id=CYPIFYENY23MG4BOVP7QV65EHGTW354LRKO4VRCXJ6C5G6IU2RAA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.596517317 (636h4m56.467600363s) [noise=true , address=45.91.116.12:4226 , piece_id=G2LGRPHRD7HA6HJHS7IMYU4XNMOSHHNXYN3GJBA6CTT3IE6ET7LQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.598204309 (636h4m56.469287344s) [noise=true , address=145.79.227.8:4330 , piece_id=AAGIVTKRUXB3ZEW5UGG6VXSRCXP5ZOO67BNPUT7OTPEB7E63TZTQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.600691746 (636h4m56.471774791s) [noise=true , address=77.90.31.10:4520 , piece_id=PYPXCHEPUFQYDOWDOZ5WDDN374KOLMTPBBQRBPHTUEYHW374UL3A, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:58.281710480 (636h4m58.152793525s) [upload=557361]: Next returned nil segment Jan 1 03:33:58.281724658 (636h4m58.152807693s) [upload=557361]: Waiting for error group managing segments... Jan 1 03:33:58.566383030 (636h4m58.437466075s) [noise=true , address=145.79.227.41:4085 , piece_id=234273FC25CEB4BGKAHDL6MMQAO6H7LNTYBTJMKAHHB35NHUWJTA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.567784098 (636h4m58.438867173s) [noise=true , address=145.79.227.15:4873 , piece_id=7BEE44HJU6KPVZW5UII3JG6SPCTCKRK4BSLSM7UANTDYMBM4VD5Q, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.583853285 (636h4m58.45493633s) [noise=true , address=145.79.227.12:4767 , piece_id=Q5MKSZLVNH4AOSJ4O7PJ27L7OWCF2XNME4UXLRQ6IGCWNN5CILHA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.591537447 (636h4m58.462620492s) [noise=true , address=145.79.227.39:4013 , piece_id=NQZ7PMFQRYM2WWBSFXJBP26FMQZY6IAWCXHMX5J4GYYKPOLBHHJQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.592304357 (636h4m58.463387392s) [noise=true , address=145.79.227.6:4257 , piece_id=X3UE3Z55HMVVFMHXITIC5SAJGIH2J5QFDYVNY66CP2VS7DFHYMIA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.599833965 (636h4m58.47091701s) [noise=true , address=145.79.227.11:4729 , piece_id=PL6XQ6DNCVGLSXLCCAWYY4OCSALV3KXMADICWEVB3I2CSMHQICZA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.601388594 (636h4m58.472471639s) [noise=true , address=145.79.227.10:4403 , piece_id=M3K5HHL7R6JRQMFUW4TCQDOZOWRGKLNYBVYQA2YD6AM37DW2L5HA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.630591339 (636h4m58.501674374s) [noise=true , address=145.79.227.8:4330 , piece_id=AAGIVTKRUXB3ZEW5UGG6VXSRCXP5ZOO67BNPUT7OTPEB7E63TZTQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.650577436 (636h4m58.521660481s) [noise=true , address=145.79.227.5:4222 , piece_id=D3XD4KVOFPOYWMRGC5UVDG3JE3LLS4TKKR6BXL6HEF2DJPGO6JSQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.916297107 (636h4m58.787380152s) [noise=true , address=45.91.116.12:4226 , piece_id=G2LGRPHRD7HA6HJHS7IMYU4XNMOSHHNXYN3GJBA6CTT3IE6ET7LQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.920798298 (636h4m58.791881333s) [noise=true , address=45.91.116.13:4156 , piece_id=MOMILYJ6A6EMTVHHO27LNNPQECIQYQ7L4OJN5OX5PKVTH2NCE3GA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.921432575 (636h4m58.792515621s) [noise=true , address=45.91.116.14:4192 , piece_id=YJVA6TFSANAM4GXNJL57DECMD5WWPI6H4L3PAUMLMEJANGGREGUA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.955733448 (636h4m58.826816493s) [noise=true , address=45.91.116.10:4081 , piece_id=NNABYWDFRZEJENTE7UPYV4UEY55E7MWFDAHQZP4TCYPYMHQ7ODBQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.955763956 (636h4m58.826846991s) [seg_pos={0 0}, upload=557361]: Segment reached optimal threshold of 13 pieces. Jan 1 03:33:58.956096580 (636h4m58.827179625s) [noise=true , address=45.91.116.11:4120 , piece_id=CYPIFYENY23MG4BOVP7QV65EHGTW354LRKO4VRCXJ6C5G6IU2RAA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Jan 1 03:33:58.956105156 (636h4m58.827188201s) [noise=true , address=45.91.116.11:4120 , piece_id=CYPIFYENY23MG4BOVP7QV65EHGTW354LRKO4VRCXJ6C5G6IU2RAA, seg_pos={0 0}, upload=557361]: Long tail cancellation due to optimal threshold Jan 1 03:33:58.956367546 (636h4m58.827450591s) [noise=true , address=77.90.31.10:4520 , piece_id=PYPXCHEPUFQYDOWDOZ5WDDN374KOLMTPBBQRBPHTUEYHW374UL3A, seg_pos={0 0}, upload=557361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Jan 1 03:33:58.956391661 (636h4m58.827474706s) [noise=true , address=77.90.31.10:4520 , piece_id=PYPXCHEPUFQYDOWDOZ5WDDN374KOLMTPBBQRBPHTUEYHW374UL3A, seg_pos={0 0}, upload=557361]: Long tail cancellation due to optimal threshold Jan 1 03:33:58.956461695 (636h4m58.82754476s) [noise=true , address=45.91.116.5:4300 , piece_id=HJKG5RL5KWVCBVP7VJ6IDAIRDSKTHED6FOIAXIOL5FF32XLKRCEQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:37472->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:37472->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37472->45.91.116.5:4300: use of closed network connection Jan 1 03:33:58.956474369 (636h4m58.827557404s) [noise=true , address=45.91.116.5:4300 , piece_id=HJKG5RL5KWVCBVP7VJ6IDAIRDSKTHED6FOIAXIOL5FF32XLKRCEQ, seg_pos={0 0}, upload=557361]: Long tail cancellation due to optimal threshold Jan 1 03:33:58.956524344 (636h4m58.827607389s) [seg_pos={0 0}, upload=557361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:33:58.956658219 (636h4m58.827741274s) [upload=557361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:33:58.956667076 (636h4m58.827750111s) [upload=557361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:33:59.288606697 (636h4m59.159689732s) [upload=557361]: Done uploading segments... Jan 1 03:33:59.288632837 (636h4m59.159715882s) [upload=557361]: Upload finished. err: <nil> Duration: 8.047943496s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X....X...................................] duration: 50.843115285s trace: https://jaeger.dev.storj.io/trace/725fc16e891a4127 FAIL (2026-01-01 03:35:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3:03:23.412681369 (635h34m23.283764404s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413214023 (635h34m23.284297058s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413504245 (635h34m23.28458728s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413821750 (635h34m23.284904785s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414124316 (635h34m23.285207351s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414417554 (635h34m23.285500589s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414689172 (635h34m23.285772207s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415023338 (635h34m23.286106373s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415341384 (635h34m23.286424419s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415659309 (635h34m23.286742354s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416184289 (635h34m23.287267324s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416571025 (635h34m23.28765406s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416998890 (635h34m23.288081925s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.417376770 (635h34m23.288459815s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.418791664 (635h34m23.289874709s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.420183384 (635h34m23.291266419s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.421563200 (635h34m23.292646246s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:24.823751878 (635h34m24.694834913s) [upload=556915]: Next returned nil segment Jan 1 03:03:24.823765594 (635h34m24.694848629s) [upload=556915]: Waiting for error group managing segments... Jan 1 03:03:25.191603537 (635h34m25.062686582s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.271741648 (635h34m25.142824683s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.278781112 (635h34m25.149864157s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.337014705 (635h34m25.20809774s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.372659678 (635h34m25.243742713s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.439670894 (635h34m25.31075394s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.447380494 (635h34m25.318463529s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.459093186 (635h34m25.330176231s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.602592395 (635h34m25.47367543s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.719366504 (635h34m25.59044954s) [noise=true , address=45.91.116.12:4230 , piece_id=WQHO3KOUTZSHBZIDUXANBVFSUKTBKEWHTZ3Z7BNGXGOFP2BMJHWA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.875237800 (635h34m25.746320845s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.143929177 (635h34m26.015012212s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.159675169 (635h34m26.030758204s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181352455 (635h34m26.05243549s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181379146 (635h34m26.052462181s) [seg_pos={0 0}, upload=556915]: Segment reached optimal threshold of 14 pieces. Jan 1 03:03:26.181716378 (635h34m26.052799413s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Pvm1SftEF3tnCEBqUn1ZmyVcLDBkR6pyegEb7mtzjdQiiqYqt): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181723281 (635h34m26.052806316s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181784638 (635h34m26.052867703s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181848530 (635h34m26.052931565s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181859921 (635h34m26.052942956s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181866474 (635h34m26.052949509s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181920486 (635h34m26.053003522s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:184g6qHdezLW3d2CKNW8GRgdzeVLQuMEhd8B8c1zEnkKNkeSTu): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181925296 (635h34m26.053008331s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181941377 (635h34m26.053024412s) [seg_pos={0 0}, upload=556915]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 03:03:26.182003695 (635h34m26.05308673s) [upload=556915]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:03:26.182009416 (635h34m26.053092461s) [upload=556915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:03:26.474026257 (635h34m26.345109292s) [upload=556915]: Done uploading segments... Jan 1 03:03:26.474049391 (635h34m26.345132427s) [upload=556915]: Upload finished. err: <nil> Duration: 50.843148538s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....................X......X............................X...] duration: 4.008526539s trace: https://jaeger.dev.storj.io/trace/2ad779a295dfac2 FAIL (2026-01-01 03:35:01 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.008555823s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XX.......X...X..X..X...X..X....XXX.X.X..XX.X.XXX..XX..XXXXX.] duration: 6.163481535s trace: https://jaeger.dev.storj.io/trace/511c0acf1687258e FAIL (2026-01-01 03:34:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: YJVA6TFSANAM4GXNJL57DECMD5WWPI6H4L3PAUMLMEJANGGREGUA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: MOMILYJ6A6EMTVHHO27LNNPQECIQYQ7L4OJN5OX5PKVTH2NCE3GA) context deadline exceeded ======LOGS====== Jan 1 03:33:56.314733719 (636h4m56.185816764s) [upload=557361]: Starting upload Jan 1 03:33:56.314856132 (636h4m56.185939177s) [upload=557361]: Done starting upload Jan 1 03:33:56.314936174 (636h4m56.186019209s) [upload=557361]: Uploading segments... Jan 1 03:33:56.315127339 (636h4m56.186210374s) [upload=557361]: Got next segment. Inline: false Jan 1 03:33:56.315162365 (636h4m56.186245411s) [upload=557361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:33:56.315167675 (636h4m56.186250721s) [upload=557361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:33:56.585484360 (636h4m56.456567405s) [seg_pos={0 0}, upload=557361]: Begin upload segment... Jan 1 03:33:56.585563410 (636h4m56.456646455s) [seg_pos={0 0}, upload=557361]: Done begin upload segment. Jan 1 03:33:56.585600531 (636h4m56.456683576s) [noise=true , address=145.79.227.6:4257 , piece_id=X3UE3Z55HMVVFMHXITIC5SAJGIH2J5QFDYVNY66CP2VS7DFHYMIA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.585748682 (636h4m56.456831728s) [noise=true , address=45.91.116.5:4300 , piece_id=HJKG5RL5KWVCBVP7VJ6IDAIRDSKTHED6FOIAXIOL5FF32XLKRCEQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.586623668 (636h4m56.457706704s) [noise=true , address=145.79.227.39:4013 , piece_id=NQZ7PMFQRYM2WWBSFXJBP26FMQZY6IAWCXHMX5J4GYYKPOLBHHJQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.587286000 (636h4m56.458369035s) [noise=true , address=45.91.116.13:4156 , piece_id=MOMILYJ6A6EMTVHHO27LNNPQECIQYQ7L4OJN5OX5PKVTH2NCE3GA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.587953671 (636h4m56.459036716s) [noise=true , address=45.91.116.14:4192 , piece_id=YJVA6TFSANAM4GXNJL57DECMD5WWPI6H4L3PAUMLMEJANGGREGUA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.588896376 (636h4m56.459979421s) [noise=true , address=145.79.227.41:4085 , piece_id=234273FC25CEB4BGKAHDL6MMQAO6H7LNTYBTJMKAHHB35NHUWJTA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.590458651 (636h4m56.461541716s) [noise=true , address=145.79.227.12:4767 , piece_id=Q5MKSZLVNH4AOSJ4O7PJ27L7OWCF2XNME4UXLRQ6IGCWNN5CILHA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.592544222 (636h4m56.463627297s) [noise=true , address=145.79.227.15:4873 , piece_id=7BEE44HJU6KPVZW5UII3JG6SPCTCKRK4BSLSM7UANTDYMBM4VD5Q, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.593386686 (636h4m56.464469731s) [noise=true , address=145.79.227.5:4222 , piece_id=D3XD4KVOFPOYWMRGC5UVDG3JE3LLS4TKKR6BXL6HEF2DJPGO6JSQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.594256092 (636h4m56.465339127s) [noise=true , address=45.91.116.10:4081 , piece_id=NNABYWDFRZEJENTE7UPYV4UEY55E7MWFDAHQZP4TCYPYMHQ7ODBQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.594836167 (636h4m56.465919202s) [noise=true , address=145.79.227.10:4403 , piece_id=M3K5HHL7R6JRQMFUW4TCQDOZOWRGKLNYBVYQA2YD6AM37DW2L5HA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.595705282 (636h4m56.466788327s) [noise=true , address=145.79.227.11:4729 , piece_id=PL6XQ6DNCVGLSXLCCAWYY4OCSALV3KXMADICWEVB3I2CSMHQICZA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.596215913 (636h4m56.467298948s) [noise=true , address=45.91.116.11:4120 , piece_id=CYPIFYENY23MG4BOVP7QV65EHGTW354LRKO4VRCXJ6C5G6IU2RAA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.596517317 (636h4m56.467600363s) [noise=true , address=45.91.116.12:4226 , piece_id=G2LGRPHRD7HA6HJHS7IMYU4XNMOSHHNXYN3GJBA6CTT3IE6ET7LQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.598204309 (636h4m56.469287344s) [noise=true , address=145.79.227.8:4330 , piece_id=AAGIVTKRUXB3ZEW5UGG6VXSRCXP5ZOO67BNPUT7OTPEB7E63TZTQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.600691746 (636h4m56.471774791s) [noise=true , address=77.90.31.10:4520 , piece_id=PYPXCHEPUFQYDOWDOZ5WDDN374KOLMTPBBQRBPHTUEYHW374UL3A, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:58.281710480 (636h4m58.152793525s) [upload=557361]: Next returned nil segment Jan 1 03:33:58.281724658 (636h4m58.152807693s) [upload=557361]: Waiting for error group managing segments... Jan 1 03:33:58.566383030 (636h4m58.437466075s) [noise=true , address=145.79.227.41:4085 , piece_id=234273FC25CEB4BGKAHDL6MMQAO6H7LNTYBTJMKAHHB35NHUWJTA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.567784098 (636h4m58.438867173s) [noise=true , address=145.79.227.15:4873 , piece_id=7BEE44HJU6KPVZW5UII3JG6SPCTCKRK4BSLSM7UANTDYMBM4VD5Q, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.583853285 (636h4m58.45493633s) [noise=true , address=145.79.227.12:4767 , piece_id=Q5MKSZLVNH4AOSJ4O7PJ27L7OWCF2XNME4UXLRQ6IGCWNN5CILHA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.591537447 (636h4m58.462620492s) [noise=true , address=145.79.227.39:4013 , piece_id=NQZ7PMFQRYM2WWBSFXJBP26FMQZY6IAWCXHMX5J4GYYKPOLBHHJQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.592304357 (636h4m58.463387392s) [noise=true , address=145.79.227.6:4257 , piece_id=X3UE3Z55HMVVFMHXITIC5SAJGIH2J5QFDYVNY66CP2VS7DFHYMIA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.599833965 (636h4m58.47091701s) [noise=true , address=145.79.227.11:4729 , piece_id=PL6XQ6DNCVGLSXLCCAWYY4OCSALV3KXMADICWEVB3I2CSMHQICZA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.601388594 (636h4m58.472471639s) [noise=true , address=145.79.227.10:4403 , piece_id=M3K5HHL7R6JRQMFUW4TCQDOZOWRGKLNYBVYQA2YD6AM37DW2L5HA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.630591339 (636h4m58.501674374s) [noise=true , address=145.79.227.8:4330 , piece_id=AAGIVTKRUXB3ZEW5UGG6VXSRCXP5ZOO67BNPUT7OTPEB7E63TZTQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.650577436 (636h4m58.521660481s) [noise=true , address=145.79.227.5:4222 , piece_id=D3XD4KVOFPOYWMRGC5UVDG3JE3LLS4TKKR6BXL6HEF2DJPGO6JSQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.916297107 (636h4m58.787380152s) [noise=true , address=45.91.116.12:4226 , piece_id=G2LGRPHRD7HA6HJHS7IMYU4XNMOSHHNXYN3GJBA6CTT3IE6ET7LQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.920798298 (636h4m58.791881333s) [noise=true , address=45.91.116.13:4156 , piece_id=MOMILYJ6A6EMTVHHO27LNNPQECIQYQ7L4OJN5OX5PKVTH2NCE3GA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.921432575 (636h4m58.792515621s) [noise=true , address=45.91.116.14:4192 , piece_id=YJVA6TFSANAM4GXNJL57DECMD5WWPI6H4L3PAUMLMEJANGGREGUA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.955733448 (636h4m58.826816493s) [noise=true , address=45.91.116.10:4081 , piece_id=NNABYWDFRZEJENTE7UPYV4UEY55E7MWFDAHQZP4TCYPYMHQ7ODBQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.955763956 (636h4m58.826846991s) [seg_pos={0 0}, upload=557361]: Segment reached optimal threshold of 13 pieces. Jan 1 03:33:58.956096580 (636h4m58.827179625s) [noise=true , address=45.91.116.11:4120 , piece_id=CYPIFYENY23MG4BOVP7QV65EHGTW354LRKO4VRCXJ6C5G6IU2RAA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Jan 1 03:33:58.956105156 (636h4m58.827188201s) [noise=true , address=45.91.116.11:4120 , piece_id=CYPIFYENY23MG4BOVP7QV65EHGTW354LRKO4VRCXJ6C5G6IU2RAA, seg_pos={0 0}, upload=557361]: Long tail cancellation due to optimal threshold Jan 1 03:33:58.956367546 (636h4m58.827450591s) [noise=true , address=77.90.31.10:4520 , piece_id=PYPXCHEPUFQYDOWDOZ5WDDN374KOLMTPBBQRBPHTUEYHW374UL3A, seg_pos={0 0}, upload=557361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Jan 1 03:33:58.956391661 (636h4m58.827474706s) [noise=true , address=77.90.31.10:4520 , piece_id=PYPXCHEPUFQYDOWDOZ5WDDN374KOLMTPBBQRBPHTUEYHW374UL3A, seg_pos={0 0}, upload=557361]: Long tail cancellation due to optimal threshold Jan 1 03:33:58.956461695 (636h4m58.82754476s) [noise=true , address=45.91.116.5:4300 , piece_id=HJKG5RL5KWVCBVP7VJ6IDAIRDSKTHED6FOIAXIOL5FF32XLKRCEQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:37472->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:37472->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37472->45.91.116.5:4300: use of closed network connection Jan 1 03:33:58.956474369 (636h4m58.827557404s) [noise=true , address=45.91.116.5:4300 , piece_id=HJKG5RL5KWVCBVP7VJ6IDAIRDSKTHED6FOIAXIOL5FF32XLKRCEQ, seg_pos={0 0}, upload=557361]: Long tail cancellation due to optimal threshold Jan 1 03:33:58.956524344 (636h4m58.827607389s) [seg_pos={0 0}, upload=557361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:33:58.956658219 (636h4m58.827741274s) [upload=557361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:33:58.956667076 (636h4m58.827750111s) [upload=557361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:33:59.288606697 (636h4m59.159689732s) [upload=557361]: Done uploading segments... Jan 1 03:33:59.288632837 (636h4m59.159715882s) [upload=557361]: Upload finished. err: <nil> Duration: 6.163510631s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [.XX.......X...X..X..X...X..X....XXX.X.X..XX.X.XXX..XX..XXXXX] duration: 30.00353164s trace: https://jaeger.dev.storj.io/trace/511c0acf1687258e FAIL (2026-01-01 03:34:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: YJVA6TFSANAM4GXNJL57DECMD5WWPI6H4L3PAUMLMEJANGGREGUA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: MOMILYJ6A6EMTVHHO27LNNPQECIQYQ7L4OJN5OX5PKVTH2NCE3GA) context deadline exceeded ======LOGS====== Jan 1 03:33:56.314733719 (636h4m56.185816764s) [upload=557361]: Starting upload Jan 1 03:33:56.314856132 (636h4m56.185939177s) [upload=557361]: Done starting upload Jan 1 03:33:56.314936174 (636h4m56.186019209s) [upload=557361]: Uploading segments... Jan 1 03:33:56.315127339 (636h4m56.186210374s) [upload=557361]: Got next segment. Inline: false Jan 1 03:33:56.315162365 (636h4m56.186245411s) [upload=557361]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:33:56.315167675 (636h4m56.186250721s) [upload=557361]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:33:56.585484360 (636h4m56.456567405s) [seg_pos={0 0}, upload=557361]: Begin upload segment... Jan 1 03:33:56.585563410 (636h4m56.456646455s) [seg_pos={0 0}, upload=557361]: Done begin upload segment. Jan 1 03:33:56.585600531 (636h4m56.456683576s) [noise=true , address=145.79.227.6:4257 , piece_id=X3UE3Z55HMVVFMHXITIC5SAJGIH2J5QFDYVNY66CP2VS7DFHYMIA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.585748682 (636h4m56.456831728s) [noise=true , address=45.91.116.5:4300 , piece_id=HJKG5RL5KWVCBVP7VJ6IDAIRDSKTHED6FOIAXIOL5FF32XLKRCEQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.586623668 (636h4m56.457706704s) [noise=true , address=145.79.227.39:4013 , piece_id=NQZ7PMFQRYM2WWBSFXJBP26FMQZY6IAWCXHMX5J4GYYKPOLBHHJQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.587286000 (636h4m56.458369035s) [noise=true , address=45.91.116.13:4156 , piece_id=MOMILYJ6A6EMTVHHO27LNNPQECIQYQ7L4OJN5OX5PKVTH2NCE3GA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.587953671 (636h4m56.459036716s) [noise=true , address=45.91.116.14:4192 , piece_id=YJVA6TFSANAM4GXNJL57DECMD5WWPI6H4L3PAUMLMEJANGGREGUA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.588896376 (636h4m56.459979421s) [noise=true , address=145.79.227.41:4085 , piece_id=234273FC25CEB4BGKAHDL6MMQAO6H7LNTYBTJMKAHHB35NHUWJTA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.590458651 (636h4m56.461541716s) [noise=true , address=145.79.227.12:4767 , piece_id=Q5MKSZLVNH4AOSJ4O7PJ27L7OWCF2XNME4UXLRQ6IGCWNN5CILHA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.592544222 (636h4m56.463627297s) [noise=true , address=145.79.227.15:4873 , piece_id=7BEE44HJU6KPVZW5UII3JG6SPCTCKRK4BSLSM7UANTDYMBM4VD5Q, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.593386686 (636h4m56.464469731s) [noise=true , address=145.79.227.5:4222 , piece_id=D3XD4KVOFPOYWMRGC5UVDG3JE3LLS4TKKR6BXL6HEF2DJPGO6JSQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.594256092 (636h4m56.465339127s) [noise=true , address=45.91.116.10:4081 , piece_id=NNABYWDFRZEJENTE7UPYV4UEY55E7MWFDAHQZP4TCYPYMHQ7ODBQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.594836167 (636h4m56.465919202s) [noise=true , address=145.79.227.10:4403 , piece_id=M3K5HHL7R6JRQMFUW4TCQDOZOWRGKLNYBVYQA2YD6AM37DW2L5HA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.595705282 (636h4m56.466788327s) [noise=true , address=145.79.227.11:4729 , piece_id=PL6XQ6DNCVGLSXLCCAWYY4OCSALV3KXMADICWEVB3I2CSMHQICZA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.596215913 (636h4m56.467298948s) [noise=true , address=45.91.116.11:4120 , piece_id=CYPIFYENY23MG4BOVP7QV65EHGTW354LRKO4VRCXJ6C5G6IU2RAA, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.596517317 (636h4m56.467600363s) [noise=true , address=45.91.116.12:4226 , piece_id=G2LGRPHRD7HA6HJHS7IMYU4XNMOSHHNXYN3GJBA6CTT3IE6ET7LQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.598204309 (636h4m56.469287344s) [noise=true , address=145.79.227.8:4330 , piece_id=AAGIVTKRUXB3ZEW5UGG6VXSRCXP5ZOO67BNPUT7OTPEB7E63TZTQ, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:56.600691746 (636h4m56.471774791s) [noise=true , address=77.90.31.10:4520 , piece_id=PYPXCHEPUFQYDOWDOZ5WDDN374KOLMTPBBQRBPHTUEYHW374UL3A, seg_pos={0 0}, upload=557361]: Uploading piece... Jan 1 03:33:58.281710480 (636h4m58.152793525s) [upload=557361]: Next returned nil segment Jan 1 03:33:58.281724658 (636h4m58.152807693s) [upload=557361]: Waiting for error group managing segments... Jan 1 03:33:58.566383030 (636h4m58.437466075s) [noise=true , address=145.79.227.41:4085 , piece_id=234273FC25CEB4BGKAHDL6MMQAO6H7LNTYBTJMKAHHB35NHUWJTA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.567784098 (636h4m58.438867173s) [noise=true , address=145.79.227.15:4873 , piece_id=7BEE44HJU6KPVZW5UII3JG6SPCTCKRK4BSLSM7UANTDYMBM4VD5Q, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.583853285 (636h4m58.45493633s) [noise=true , address=145.79.227.12:4767 , piece_id=Q5MKSZLVNH4AOSJ4O7PJ27L7OWCF2XNME4UXLRQ6IGCWNN5CILHA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.591537447 (636h4m58.462620492s) [noise=true , address=145.79.227.39:4013 , piece_id=NQZ7PMFQRYM2WWBSFXJBP26FMQZY6IAWCXHMX5J4GYYKPOLBHHJQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.592304357 (636h4m58.463387392s) [noise=true , address=145.79.227.6:4257 , piece_id=X3UE3Z55HMVVFMHXITIC5SAJGIH2J5QFDYVNY66CP2VS7DFHYMIA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.599833965 (636h4m58.47091701s) [noise=true , address=145.79.227.11:4729 , piece_id=PL6XQ6DNCVGLSXLCCAWYY4OCSALV3KXMADICWEVB3I2CSMHQICZA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.601388594 (636h4m58.472471639s) [noise=true , address=145.79.227.10:4403 , piece_id=M3K5HHL7R6JRQMFUW4TCQDOZOWRGKLNYBVYQA2YD6AM37DW2L5HA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.630591339 (636h4m58.501674374s) [noise=true , address=145.79.227.8:4330 , piece_id=AAGIVTKRUXB3ZEW5UGG6VXSRCXP5ZOO67BNPUT7OTPEB7E63TZTQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.650577436 (636h4m58.521660481s) [noise=true , address=145.79.227.5:4222 , piece_id=D3XD4KVOFPOYWMRGC5UVDG3JE3LLS4TKKR6BXL6HEF2DJPGO6JSQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.916297107 (636h4m58.787380152s) [noise=true , address=45.91.116.12:4226 , piece_id=G2LGRPHRD7HA6HJHS7IMYU4XNMOSHHNXYN3GJBA6CTT3IE6ET7LQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.920798298 (636h4m58.791881333s) [noise=true , address=45.91.116.13:4156 , piece_id=MOMILYJ6A6EMTVHHO27LNNPQECIQYQ7L4OJN5OX5PKVTH2NCE3GA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.921432575 (636h4m58.792515621s) [noise=true , address=45.91.116.14:4192 , piece_id=YJVA6TFSANAM4GXNJL57DECMD5WWPI6H4L3PAUMLMEJANGGREGUA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.955733448 (636h4m58.826816493s) [noise=true , address=45.91.116.10:4081 , piece_id=NNABYWDFRZEJENTE7UPYV4UEY55E7MWFDAHQZP4TCYPYMHQ7ODBQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: <nil> Jan 1 03:33:58.955763956 (636h4m58.826846991s) [seg_pos={0 0}, upload=557361]: Segment reached optimal threshold of 13 pieces. Jan 1 03:33:58.956096580 (636h4m58.827179625s) [noise=true , address=45.91.116.11:4120 , piece_id=CYPIFYENY23MG4BOVP7QV65EHGTW354LRKO4VRCXJ6C5G6IU2RAA, seg_pos={0 0}, upload=557361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF Jan 1 03:33:58.956105156 (636h4m58.827188201s) [noise=true , address=45.91.116.11:4120 , piece_id=CYPIFYENY23MG4BOVP7QV65EHGTW354LRKO4VRCXJ6C5G6IU2RAA, seg_pos={0 0}, upload=557361]: Long tail cancellation due to optimal threshold Jan 1 03:33:58.956367546 (636h4m58.827450591s) [noise=true , address=77.90.31.10:4520 , piece_id=PYPXCHEPUFQYDOWDOZ5WDDN374KOLMTPBBQRBPHTUEYHW374UL3A, seg_pos={0 0}, upload=557361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Jan 1 03:33:58.956391661 (636h4m58.827474706s) [noise=true , address=77.90.31.10:4520 , piece_id=PYPXCHEPUFQYDOWDOZ5WDDN374KOLMTPBBQRBPHTUEYHW374UL3A, seg_pos={0 0}, upload=557361]: Long tail cancellation due to optimal threshold Jan 1 03:33:58.956461695 (636h4m58.82754476s) [noise=true , address=45.91.116.5:4300 , piece_id=HJKG5RL5KWVCBVP7VJ6IDAIRDSKTHED6FOIAXIOL5FF32XLKRCEQ, seg_pos={0 0}, upload=557361]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:37472->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:37472->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37472->45.91.116.5:4300: use of closed network connection Jan 1 03:33:58.956474369 (636h4m58.827557404s) [noise=true , address=45.91.116.5:4300 , piece_id=HJKG5RL5KWVCBVP7VJ6IDAIRDSKTHED6FOIAXIOL5FF32XLKRCEQ, seg_pos={0 0}, upload=557361]: Long tail cancellation due to optimal threshold Jan 1 03:33:58.956524344 (636h4m58.827607389s) [seg_pos={0 0}, upload=557361]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:33:58.956658219 (636h4m58.827741274s) [upload=557361]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:33:58.956667076 (636h4m58.827750111s) [upload=557361]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:33:59.288606697 (636h4m59.159689732s) [upload=557361]: Done uploading segments... Jan 1 03:33:59.288632837 (636h4m59.159715882s) [upload=557361]: Upload finished. err: <nil> Duration: 30.003575052s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....................X......X............................X..] duration: 4.024954559s trace: https://jaeger.dev.storj.io/trace/2ad779a295dfac2 FAIL (2026-01-01 03:33:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.025094193s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X....X..................................] duration: 34.105672381s trace: https://jaeger.dev.storj.io/trace/725fc16e891a4127 FAIL (2026-01-01 03:33:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3:03:23.412681369 (635h34m23.283764404s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413214023 (635h34m23.284297058s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413504245 (635h34m23.28458728s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413821750 (635h34m23.284904785s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414124316 (635h34m23.285207351s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414417554 (635h34m23.285500589s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414689172 (635h34m23.285772207s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415023338 (635h34m23.286106373s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415341384 (635h34m23.286424419s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415659309 (635h34m23.286742354s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416184289 (635h34m23.287267324s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416571025 (635h34m23.28765406s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416998890 (635h34m23.288081925s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.417376770 (635h34m23.288459815s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.418791664 (635h34m23.289874709s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.420183384 (635h34m23.291266419s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.421563200 (635h34m23.292646246s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:24.823751878 (635h34m24.694834913s) [upload=556915]: Next returned nil segment Jan 1 03:03:24.823765594 (635h34m24.694848629s) [upload=556915]: Waiting for error group managing segments... Jan 1 03:03:25.191603537 (635h34m25.062686582s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.271741648 (635h34m25.142824683s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.278781112 (635h34m25.149864157s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.337014705 (635h34m25.20809774s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.372659678 (635h34m25.243742713s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.439670894 (635h34m25.31075394s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.447380494 (635h34m25.318463529s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.459093186 (635h34m25.330176231s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.602592395 (635h34m25.47367543s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.719366504 (635h34m25.59044954s) [noise=true , address=45.91.116.12:4230 , piece_id=WQHO3KOUTZSHBZIDUXANBVFSUKTBKEWHTZ3Z7BNGXGOFP2BMJHWA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.875237800 (635h34m25.746320845s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.143929177 (635h34m26.015012212s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.159675169 (635h34m26.030758204s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181352455 (635h34m26.05243549s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181379146 (635h34m26.052462181s) [seg_pos={0 0}, upload=556915]: Segment reached optimal threshold of 14 pieces. Jan 1 03:03:26.181716378 (635h34m26.052799413s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Pvm1SftEF3tnCEBqUn1ZmyVcLDBkR6pyegEb7mtzjdQiiqYqt): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181723281 (635h34m26.052806316s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181784638 (635h34m26.052867703s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181848530 (635h34m26.052931565s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181859921 (635h34m26.052942956s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181866474 (635h34m26.052949509s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181920486 (635h34m26.053003522s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:184g6qHdezLW3d2CKNW8GRgdzeVLQuMEhd8B8c1zEnkKNkeSTu): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181925296 (635h34m26.053008331s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181941377 (635h34m26.053024412s) [seg_pos={0 0}, upload=556915]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 03:03:26.182003695 (635h34m26.05308673s) [upload=556915]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:03:26.182009416 (635h34m26.053092461s) [upload=556915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:03:26.474026257 (635h34m26.345109292s) [upload=556915]: Done uploading segments... Jan 1 03:03:26.474049391 (635h34m26.345132427s) [upload=556915]: Upload finished. err: <nil> Duration: 34.10570254s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [X.XX.......X...X..X..X...X..X....XXX.X.X..XX.X.XXX..XX..XXXX] duration: 30.00467481s trace: https://jaeger.dev.storj.io/trace/580b40c83d5cd86b FAIL (2026-01-01 03:33:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: RNYWJ7V2FXFR2TSJQA74MJDOBA6N3ELXXUNT3SIQZBSE7SL72W2Q) context deadline exceeded; piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: MFSZBF6TEYAXDEPNRNBSN7W74S6KI2OYSHJOOMMXPZTMSLJHC4SQ) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: 5OLGIAFDSEPAE3SOYELXSCQU3TXI4NYTDSBW6ADWQV76AU5NG5CQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: VGH4WC5WMQ54736XZ2PDKMHS27VQ7J6CBHZI3TV6NKHJP3G54CVA) context deadline exceeded ======LOGS====== Jan 1 03:32:55.709775520 (636h3m55.580858565s) [upload=557348]: Starting upload Jan 1 03:32:55.709860711 (636h3m55.580943757s) [upload=557348]: Done starting upload Jan 1 03:32:55.709946325 (636h3m55.58102937s) [upload=557348]: Uploading segments... Jan 1 03:32:55.710112120 (636h3m55.581195155s) [upload=557348]: Got next segment. Inline: false Jan 1 03:32:55.710148549 (636h3m55.581231584s) [upload=557348]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:32:55.710153560 (636h3m55.581236595s) [upload=557348]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:32:55.955853521 (636h3m55.826936566s) [seg_pos={0 0}, upload=557348]: Begin upload segment... Jan 1 03:32:55.955911430 (636h3m55.826994465s) [seg_pos={0 0}, upload=557348]: Done begin upload segment. Jan 1 03:32:55.955957579 (636h3m55.827040614s) [noise=true , address=145.79.227.18:4006 , piece_id=OV2GBZFSO34TJRBFVY3ZY6OEESSCEUES3YYKFIGZVE3TSDNZG4AA, seg_pos={0 0}, upload=557348]: Uploading piece... Jan 1 03:32:55.955992275 (636h3m55.82707531s) [noise=true , address=77.90.31.21:4880 , piece_id=RNYWJ7V2FXFR2TSJQA74MJDOBA6N3ELXXUNT3SIQZBSE7SL72W2Q, seg_pos={0 0}, upload=557348]: Uploading piece... Jan 1 03:32:55.956054934 (636h3m55.827137979s) [noise=true , address=145.79.227.52:4447 , piece_id=VROQ7OP7TQ6BU53NQNDPRVZ4DZPLOLRWUPRWBHRJDJH2YG7HNERA, seg_pos={0 0}, upload=557348]: Uploading piece... Jan 1 03:32:55.956646029 (636h3m55.827729064s) [noise=true , address=145.79.227.37:4565 , piece_id=GRVTGGMC66ULVCMTFZTIZJUEW3ZD7PR6UYJTHM3YPIQK6X5UJSDQ, seg_pos={0 0}, upload=557348]: Uploading piece... Jan 1 03:32:55.956777540 (636h3m55.827860585s) [noise=true , address=45.91.116.9:4264 , piece_id=VVK3P3Y4M5Y3V4FBCOUXB7R4EMYGV4Q7SCX6GBONJ7HHSAAL5CMQ, seg_pos={0 0}, upload=557348]: Uploading piece... Jan 1 03:32:55.957448437 (636h3m55.828531472s) [noise=true , address=145.79.227.2:4115 , piece_id=4BS4DSC7HIGFQQF42772655RMCPBC3Y2MTD7MRTYELCR6JE765EA, seg_pos={0 0}, upload=557348]: Uploading piece... Jan 1 03:32:55.958688408 (636h3m55.829771443s) [noise=true , address=145.79.227.10:4403 , piece_id=NG4P6GSE5X7AKAEQ6N6GM4ZAWDFJ7EWVNEFZZ4DOCOAVINQEMYBA, seg_pos={0 0}, upload=557348]: Uploading piece... Jan 1 03:32:55.959635582 (636h3m55.830718617s) [noise=true , address=77.90.31.13:4627 , piece_id=MFSZBF6TEYAXDEPNRNBSN7W74S6KI2OYSHJOOMMXPZTMSLJHC4SQ, seg_pos={0 0}, upload=557348]: Uploading piece... Jan 1 03:32:55.960200277 (636h3m55.831283322s) [noise=true , address=45.91.116.14:4190 , piece_id=VGH4WC5WMQ54736XZ2PDKMHS27VQ7J6CBHZI3TV6NKHJP3G54CVA, seg_pos={0 0}, upload=557348]: Uploading piece... Jan 1 03:32:55.960643611 (636h3m55.831726656s) [noise=true , address=145.79.227.38:4977 , piece_id=BX7TGUH7FPQWR3IGWYF5FW76TEJ6ZD3P3RR5KZEC7363R2RQ5GIQ, seg_pos={0 0}, upload=557348]: Uploading piece... Jan 1 03:32:55.961244365 (636h3m55.83232741s) [noise=true , address=77.90.31.17:4773 , piece_id=YMMIFWC3IKTPKPM3XVPOVTLOPFJGIY6TNSUJQPWIMBGVKN6OWT7Q, seg_pos={0 0}, upload=557348]: Uploading piece... Jan 1 03:32:55.961570465 (636h3m55.832653511s) [noise=true , address=145.79.227.20:4438 , piece_id=YR7QBFD7ICGTZ6YUXQB37ZLSG2FG46G6K235UOQ4AY6GCKCYEM6Q, seg_pos={0 0}, upload=557348]: Uploading piece... Jan 1 03:32:55.961887379 (636h3m55.832970414s) [noise=true , address=145.79.227.33:4870 , piece_id=R3HM5OE54LCBKOLXQB3YAGMIEHNS2736PGVZNOE32IF36G6FAX5A, seg_pos={0 0}, upload=557348]: Uploading piece... Jan 1 03:32:55.962068344 (636h3m55.833151379s) [noise=true , address=45.91.116.12:4226 , piece_id=5OLGIAFDSEPAE3SOYELXSCQU3TXI4NYTDSBW6ADWQV76AU5NG5CQ, seg_pos={0 0}, upload=557348]: Uploading piece... Jan 1 03:32:55.962943641 (636h3m55.834026676s) [noise=true , address=77.90.31.10:4518 , piece_id=G52XWUBOIH6MBY4QMNNXV43TNU243QLFNOVXYHEE5WSCQGM5EU4Q, seg_pos={0 0}, upload=557348]: Uploading piece... Jan 1 03:32:55.963182885 (636h3m55.834265921s) [noise=true , address=145.79.227.45:4230 , piece_id=7E4AHPDT2TPTQX5NA25JTWMXOWU4FN3ZJDCTB5G5O5OS4LB5K2RQ, seg_pos={0 0}, upload=557348]: Uploading piece... Jan 1 03:32:57.652445537 (636h3m57.523528582s) [upload=557348]: Next returned nil segment Jan 1 03:32:57.652484081 (636h3m57.523567116s) [upload=557348]: Waiting for error group managing segments... Jan 1 03:32:57.936564262 (636h3m57.807647297s) [noise=true , address=145.79.227.45:4230 , piece_id=7E4AHPDT2TPTQX5NA25JTWMXOWU4FN3ZJDCTB5G5O5OS4LB5K2RQ, seg_pos={0 0}, upload=557348]: Done uploading piece. err: <nil> Jan 1 03:32:57.944531322 (636h3m57.815614368s) [noise=true , address=145.79.227.10:4403 , piece_id=NG4P6GSE5X7AKAEQ6N6GM4ZAWDFJ7EWVNEFZZ4DOCOAVINQEMYBA, seg_pos={0 0}, upload=557348]: Done uploading piece. err: <nil> Jan 1 03:32:57.952735855 (636h3m57.8238189s) [noise=true , address=145.79.227.37:4565 , piece_id=GRVTGGMC66ULVCMTFZTIZJUEW3ZD7PR6UYJTHM3YPIQK6X5UJSDQ, seg_pos={0 0}, upload=557348]: Done uploading piece. err: <nil> Jan 1 03:32:57.957181489 (636h3m57.828264524s) [noise=true , address=145.79.227.20:4438 , piece_id=YR7QBFD7ICGTZ6YUXQB37ZLSG2FG46G6K235UOQ4AY6GCKCYEM6Q, seg_pos={0 0}, upload=557348]: Done uploading piece. err: <nil> Jan 1 03:32:57.971171138 (636h3m57.842254173s) [noise=true , address=145.79.227.52:4447 , piece_id=VROQ7OP7TQ6BU53NQNDPRVZ4DZPLOLRWUPRWBHRJDJH2YG7HNERA, seg_pos={0 0}, upload=557348]: Done uploading piece. err: <nil> Jan 1 03:32:57.973415100 (636h3m57.844498145s) [noise=true , address=145.79.227.2:4115 , piece_id=4BS4DSC7HIGFQQF42772655RMCPBC3Y2MTD7MRTYELCR6JE765EA, seg_pos={0 0}, upload=557348]: Done uploading piece. err: <nil> Jan 1 03:32:57.980749576 (636h3m57.851832611s) [noise=true , address=145.79.227.38:4977 , piece_id=BX7TGUH7FPQWR3IGWYF5FW76TEJ6ZD3P3RR5KZEC7363R2RQ5GIQ, seg_pos={0 0}, upload=557348]: Done uploading piece. err: <nil> Jan 1 03:32:58.029969767 (636h3m57.901052812s) [noise=true , address=145.79.227.18:4006 , piece_id=OV2GBZFSO34TJRBFVY3ZY6OEESSCEUES3YYKFIGZVE3TSDNZG4AA, seg_pos={0 0}, upload=557348]: Done uploading piece. err: <nil> Jan 1 03:32:58.154161673 (636h3m58.025244728s) [noise=true , address=77.90.31.13:4627 , piece_id=MFSZBF6TEYAXDEPNRNBSN7W74S6KI2OYSHJOOMMXPZTMSLJHC4SQ, seg_pos={0 0}, upload=557348]: Done uploading piece. err: <nil> Jan 1 03:32:58.232702424 (636h3m58.103785459s) [noise=true , address=77.90.31.10:4518 , piece_id=G52XWUBOIH6MBY4QMNNXV43TNU243QLFNOVXYHEE5WSCQGM5EU4Q, seg_pos={0 0}, upload=557348]: Done uploading piece. err: <nil> Jan 1 03:32:58.240396674 (636h3m58.111479719s) [noise=true , address=77.90.31.21:4880 , piece_id=RNYWJ7V2FXFR2TSJQA74MJDOBA6N3ELXXUNT3SIQZBSE7SL72W2Q, seg_pos={0 0}, upload=557348]: Done uploading piece. err: <nil> Jan 1 03:32:59.027232650 (636h3m58.898315695s) [noise=true , address=45.91.116.14:4190 , piece_id=VGH4WC5WMQ54736XZ2PDKMHS27VQ7J6CBHZI3TV6NKHJP3G54CVA, seg_pos={0 0}, upload=557348]: Done uploading piece. err: <nil> Jan 1 03:32:59.297714539 (636h3m59.168797585s) [noise=true , address=45.91.116.12:4226 , piece_id=5OLGIAFDSEPAE3SOYELXSCQU3TXI4NYTDSBW6ADWQV76AU5NG5CQ, seg_pos={0 0}, upload=557348]: Done uploading piece. err: <nil> Jan 1 03:32:59.297741691 (636h3m59.168824736s) [seg_pos={0 0}, upload=557348]: Segment reached optimal threshold of 13 pieces. Jan 1 03:32:59.298014130 (636h3m59.169097185s) [noise=true , address=145.79.227.33:4870 , piece_id=R3HM5OE54LCBKOLXQB3YAGMIEHNS2736PGVZNOE32IF36G6FAX5A, seg_pos={0 0}, upload=557348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Jan 1 03:32:59.298022736 (636h3m59.169105772s) [noise=true , address=145.79.227.33:4870 , piece_id=R3HM5OE54LCBKOLXQB3YAGMIEHNS2736PGVZNOE32IF36G6FAX5A, seg_pos={0 0}, upload=557348]: Long tail cancellation due to optimal threshold Jan 1 03:32:59.298335512 (636h3m59.169418547s) [noise=true , address=45.91.116.9:4264 , piece_id=VVK3P3Y4M5Y3V4FBCOUXB7R4EMYGV4Q7SCX6GBONJ7HHSAAL5CMQ, seg_pos={0 0}, upload=557348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Jan 1 03:32:59.298343317 (636h3m59.169426362s) [noise=true , address=45.91.116.9:4264 , piece_id=VVK3P3Y4M5Y3V4FBCOUXB7R4EMYGV4Q7SCX6GBONJ7HHSAAL5CMQ, seg_pos={0 0}, upload=557348]: Long tail cancellation due to optimal threshold Jan 1 03:32:59.298498241 (636h3m59.169581286s) [noise=true , address=77.90.31.17:4773 , piece_id=YMMIFWC3IKTPKPM3XVPOVTLOPFJGIY6TNSUJQPWIMBGVKN6OWT7Q, seg_pos={0 0}, upload=557348]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Jan 1 03:32:59.298507059 (636h3m59.169590104s) [noise=true , address=77.90.31.17:4773 , piece_id=YMMIFWC3IKTPKPM3XVPOVTLOPFJGIY6TNSUJQPWIMBGVKN6OWT7Q, seg_pos={0 0}, upload=557348]: Long tail cancellation due to optimal threshold Jan 1 03:32:59.298525083 (636h3m59.169608118s) [seg_pos={0 0}, upload=557348]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:32:59.298601798 (636h3m59.169684834s) [upload=557348]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:32:59.298607400 (636h3m59.169690445s) [upload=557348]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:32:59.596195152 (636h3m59.467278187s) [upload=557348]: Done uploading segments... Jan 1 03:32:59.596215400 (636h3m59.467298435s) [upload=557348]: Upload finished. err: <nil> Duration: 30.004700479s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......................X......X............................X.] duration: 3.883828911s trace: https://jaeger.dev.storj.io/trace/2ad779a295dfac2 FAIL (2026-01-01 03:33:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.883858386s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.X.XX.......X...X..X..X...X..X....XXX.X.X..XX.X.XXX..XX..XXX] duration: 30.003225706s trace: https://jaeger.dev.storj.io/trace/5f996ce9d9bbf2fe FAIL (2026-01-01 03:32:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: KYZOR5QPXYNGF2QPALDVMMCKOWE67JN3GCGHCRCFPEFOQAGODBFA) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: 6AK7HHAD2I45VLTX7T2IHLAJCNIJJE6VMBNLSPENYMNKCTVQQRVQ) context deadline exceeded ======LOGS====== Jan 1 03:32:21.533126069 (636h3m21.404209104s) [upload=557340]: Starting upload Jan 1 03:32:21.533183809 (636h3m21.404266844s) [upload=557340]: Done starting upload Jan 1 03:32:21.533305360 (636h3m21.404388405s) [upload=557340]: Uploading segments... Jan 1 03:32:21.533429076 (636h3m21.404512121s) [upload=557340]: Got next segment. Inline: false Jan 1 03:32:21.533475845 (636h3m21.40455889s) [upload=557340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:32:21.533481676 (636h3m21.404564721s) [upload=557340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:32:21.779280165 (636h3m21.65036321s) [seg_pos={0 0}, upload=557340]: Begin upload segment... Jan 1 03:32:21.779373954 (636h3m21.650456989s) [seg_pos={0 0}, upload=557340]: Done begin upload segment. Jan 1 03:32:21.779471890 (636h3m21.650554935s) [noise=true , address=45.91.116.11:4120 , piece_id=KYZOR5QPXYNGF2QPALDVMMCKOWE67JN3GCGHCRCFPEFOQAGODBFA, seg_pos={0 0}, upload=557340]: Uploading piece... Jan 1 03:32:21.780649683 (636h3m21.651732728s) [noise=true , address=145.79.227.26:4653 , piece_id=NLQPCKTXXOHB6IOCIYO2EMBBH6O436ISSK7WFVF4T752XJMAH55Q, seg_pos={0 0}, upload=557340]: Uploading piece... Jan 1 03:32:21.780710608 (636h3m21.651793654s) [noise=true , address=145.79.227.44:4193 , piece_id=CFF6YPJVIP7FLNF6VXBZTX5C6KAYARTBRYLJASE2YGD43OVPN4LA, seg_pos={0 0}, upload=557340]: Uploading piece... Jan 1 03:32:21.781433004 (636h3m21.652516049s) [noise=true , address=145.79.227.49:4339 , piece_id=TJW3ZHTOIQS2NLIYOQQLUEFEMNK3MTP4HFVH442JEB7DWRXENFTQ, seg_pos={0 0}, upload=557340]: Uploading piece... Jan 1 03:32:21.781554156 (636h3m21.652637211s) [noise=true , address=145.79.227.36:4518 , piece_id=SDJPCHG4AFP6RLNN3OZ2RIHSYKWGYJA6ZK4ZGMXFVNTIY2IL6PMA, seg_pos={0 0}, upload=557340]: Uploading piece... Jan 1 03:32:21.782056361 (636h3m21.653139396s) [noise=true , address=145.79.227.15:4876 , piece_id=MIUFMGPY23NPR2Z4K4QVYDJDY3S5YHONS3YRMG6VNECKIYGM4KTA, seg_pos={0 0}, upload=557340]: Uploading piece... Jan 1 03:32:21.782066911 (636h3m21.653149946s) [noise=true , address=45.91.116.14:4189 , piece_id=IE4BVWR2A4WQFSUPM5Q2FFZHSSBS575G4VKOZJ7NBPXTD442CDJQ, seg_pos={0 0}, upload=557340]: Uploading piece... Jan 1 03:32:21.782459008 (636h3m21.653542043s) [noise=true , address=145.79.227.48:4301 , piece_id=4SPOX6OMJ7PBI4GI3INKON7F5UYCASAHYCZ4ZZRLBNJLVPP5N52A, seg_pos={0 0}, upload=557340]: Uploading piece... Jan 1 03:32:21.782513662 (636h3m21.653596707s) [noise=true , address=45.91.116.5:4297 , piece_id=ZPKLTUMT3WQ2ILHUU5XBR5HYKZKUWEUVF6ZTVXPO4YBXXK4P5A3A, seg_pos={0 0}, upload=557340]: Uploading piece... Jan 1 03:32:21.782813423 (636h3m21.653896468s) [noise=true , address=77.90.31.19:4842 , piece_id=6AK7HHAD2I45VLTX7T2IHLAJCNIJJE6VMBNLSPENYMNKCTVQQRVQ, seg_pos={0 0}, upload=557340]: Uploading piece... Jan 1 03:32:21.783429045 (636h3m21.65451208s) [noise=true , address=145.79.227.42:4122 , piece_id=NCYULIDBIUPKJ5PUZQH6JKZKQZOX4WDCDQUAQYQDH2UTL527TYCQ, seg_pos={0 0}, upload=557340]: Uploading piece... Jan 1 03:32:21.783977630 (636h3m21.655060675s) [noise=true , address=145.79.227.38:4977 , piece_id=ASUYRO6S4ZCQXNPWQ5E5LIILESZAMH3YFUGKPLPFODJXR5QWKZPQ, seg_pos={0 0}, upload=557340]: Uploading piece... Jan 1 03:32:21.784379925 (636h3m21.65546297s) [noise=true , address=145.79.227.14:4839 , piece_id=5W3RQKD25LDVZB3DBAJZUEKBITBAXIL2EEXXIM3WRAKCMHAQQVKA, seg_pos={0 0}, upload=557340]: Uploading piece... Jan 1 03:32:21.784673465 (636h3m21.6557565s) [noise=true , address=145.79.227.24:4583 , piece_id=6274FGXZGVND4OZ6J5IBIFHZEMIJ5WIPTDS3D4SJ6NLZQBINV36A, seg_pos={0 0}, upload=557340]: Uploading piece... Jan 1 03:32:21.785140574 (636h3m21.656223619s) [noise=true , address=145.79.227.10:4401 , piece_id=B7JG6XH6ULIR4GM34QXQXPIWMADYHAGG5APXPYYCGXNEQ2UZV7FA, seg_pos={0 0}, upload=557340]: Uploading piece... Jan 1 03:32:21.786476729 (636h3m21.657559774s) [noise=true , address=145.79.227.17:4972 , piece_id=LDRLCVE5WL2UMRUQSCLQPLZCNVK3YR4CKEAYO27QLFOHZ3YLQDQA, seg_pos={0 0}, upload=557340]: Uploading piece... Jan 1 03:32:23.492415864 (636h3m23.363498909s) [upload=557340]: Next returned nil segment Jan 1 03:32:23.492442043 (636h3m23.363525078s) [upload=557340]: Waiting for error group managing segments... Jan 1 03:32:23.766492066 (636h3m23.637575101s) [noise=true , address=145.79.227.10:4401 , piece_id=B7JG6XH6ULIR4GM34QXQXPIWMADYHAGG5APXPYYCGXNEQ2UZV7FA, seg_pos={0 0}, upload=557340]: Done uploading piece. err: <nil> Jan 1 03:32:23.782627920 (636h3m23.653710965s) [noise=true , address=145.79.227.42:4122 , piece_id=NCYULIDBIUPKJ5PUZQH6JKZKQZOX4WDCDQUAQYQDH2UTL527TYCQ, seg_pos={0 0}, upload=557340]: Done uploading piece. err: <nil> Jan 1 03:32:23.785317922 (636h3m23.656400967s) [noise=true , address=145.79.227.15:4876 , piece_id=MIUFMGPY23NPR2Z4K4QVYDJDY3S5YHONS3YRMG6VNECKIYGM4KTA, seg_pos={0 0}, upload=557340]: Done uploading piece. err: <nil> Jan 1 03:32:23.785507564 (636h3m23.656590599s) [noise=true , address=145.79.227.48:4301 , piece_id=4SPOX6OMJ7PBI4GI3INKON7F5UYCASAHYCZ4ZZRLBNJLVPP5N52A, seg_pos={0 0}, upload=557340]: Done uploading piece. err: <nil> Jan 1 03:32:23.788764124 (636h3m23.659847159s) [noise=true , address=145.79.227.26:4653 , piece_id=NLQPCKTXXOHB6IOCIYO2EMBBH6O436ISSK7WFVF4T752XJMAH55Q, seg_pos={0 0}, upload=557340]: Done uploading piece. err: <nil> Jan 1 03:32:23.795590282 (636h3m23.666673328s) [noise=true , address=145.79.227.17:4972 , piece_id=LDRLCVE5WL2UMRUQSCLQPLZCNVK3YR4CKEAYO27QLFOHZ3YLQDQA, seg_pos={0 0}, upload=557340]: Done uploading piece. err: <nil> Jan 1 03:32:23.809767047 (636h3m23.680850092s) [noise=true , address=145.79.227.38:4977 , piece_id=ASUYRO6S4ZCQXNPWQ5E5LIILESZAMH3YFUGKPLPFODJXR5QWKZPQ, seg_pos={0 0}, upload=557340]: Done uploading piece. err: <nil> Jan 1 03:32:23.815132945 (636h3m23.68621599s) [noise=true , address=145.79.227.14:4839 , piece_id=5W3RQKD25LDVZB3DBAJZUEKBITBAXIL2EEXXIM3WRAKCMHAQQVKA, seg_pos={0 0}, upload=557340]: Done uploading piece. err: <nil> Jan 1 03:32:23.833498404 (636h3m23.704581449s) [noise=true , address=145.79.227.24:4583 , piece_id=6274FGXZGVND4OZ6J5IBIFHZEMIJ5WIPTDS3D4SJ6NLZQBINV36A, seg_pos={0 0}, upload=557340]: Done uploading piece. err: <nil> Jan 1 03:32:23.860339824 (636h3m23.731422869s) [noise=true , address=145.79.227.36:4518 , piece_id=SDJPCHG4AFP6RLNN3OZ2RIHSYKWGYJA6ZK4ZGMXFVNTIY2IL6PMA, seg_pos={0 0}, upload=557340]: Done uploading piece. err: <nil> Jan 1 03:32:23.975885265 (636h3m23.8469683s) [noise=true , address=77.90.31.19:4842 , piece_id=6AK7HHAD2I45VLTX7T2IHLAJCNIJJE6VMBNLSPENYMNKCTVQQRVQ, seg_pos={0 0}, upload=557340]: Done uploading piece. err: <nil> Jan 1 03:32:24.136711647 (636h3m24.007794682s) [noise=true , address=45.91.116.14:4189 , piece_id=IE4BVWR2A4WQFSUPM5Q2FFZHSSBS575G4VKOZJ7NBPXTD442CDJQ, seg_pos={0 0}, upload=557340]: Done uploading piece. err: <nil> Jan 1 03:32:24.169079149 (636h3m24.040162184s) [noise=true , address=45.91.116.11:4120 , piece_id=KYZOR5QPXYNGF2QPALDVMMCKOWE67JN3GCGHCRCFPEFOQAGODBFA, seg_pos={0 0}, upload=557340]: Done uploading piece. err: <nil> Jan 1 03:32:24.169109726 (636h3m24.040192771s) [seg_pos={0 0}, upload=557340]: Segment reached optimal threshold of 13 pieces. Jan 1 03:32:24.169358230 (636h3m24.040441285s) [noise=true , address=45.91.116.5:4297 , piece_id=ZPKLTUMT3WQ2ILHUU5XBR5HYKZKUWEUVF6ZTVXPO4YBXXK4P5A3A, seg_pos={0 0}, upload=557340]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Jan 1 03:32:24.169367537 (636h3m24.040450573s) [noise=true , address=45.91.116.5:4297 , piece_id=ZPKLTUMT3WQ2ILHUU5XBR5HYKZKUWEUVF6ZTVXPO4YBXXK4P5A3A, seg_pos={0 0}, upload=557340]: Long tail cancellation due to optimal threshold Jan 1 03:32:24.169486323 (636h3m24.040569368s) [noise=true , address=145.79.227.44:4193 , piece_id=CFF6YPJVIP7FLNF6VXBZTX5C6KAYARTBRYLJASE2YGD43OVPN4LA, seg_pos={0 0}, upload=557340]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Jan 1 03:32:24.169544254 (636h3m24.040627289s) [noise=true , address=145.79.227.44:4193 , piece_id=CFF6YPJVIP7FLNF6VXBZTX5C6KAYARTBRYLJASE2YGD43OVPN4LA, seg_pos={0 0}, upload=557340]: Long tail cancellation due to optimal threshold Jan 1 03:32:24.169725509 (636h3m24.040808554s) [noise=true , address=145.79.227.49:4339 , piece_id=TJW3ZHTOIQS2NLIYOQQLUEFEMNK3MTP4HFVH442JEB7DWRXENFTQ, seg_pos={0 0}, upload=557340]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Jan 1 03:32:24.169735307 (636h3m24.040818352s) [noise=true , address=145.79.227.49:4339 , piece_id=TJW3ZHTOIQS2NLIYOQQLUEFEMNK3MTP4HFVH442JEB7DWRXENFTQ, seg_pos={0 0}, upload=557340]: Long tail cancellation due to optimal threshold Jan 1 03:32:24.169762139 (636h3m24.040845174s) [seg_pos={0 0}, upload=557340]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:32:24.169888510 (636h3m24.040971555s) [upload=557340]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:32:24.169895583 (636h3m24.040978628s) [upload=557340]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:32:24.466277248 (636h3m24.337360283s) [upload=557340]: Done uploading segments... Jan 1 03:32:24.466294611 (636h3m24.337377646s) [upload=557340]: Upload finished. err: <nil> Duration: 30.003244511s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......................X......X............................X] duration: 5.001216178s trace: https://jaeger.dev.storj.io/trace/2ad779a295dfac2 FAIL (2026-01-01 03:32:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001240835s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..X.XX.......X...X..X..X...X..X....XXX.X.X..XX.X.XXX..XX..XX] duration: 30.003870766s trace: https://jaeger.dev.storj.io/trace/27e7b1a304ec007a FAIL (2026-01-01 03:32:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: HEAZI373ADYHLWCH64NG55TMGOC5QD2VJSFH5LIBEOVQAJSNQOXQ) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 7OFTTAHEMU4YGOYH4AAVT57QPZFQ3GB2CIW6374MV3UOYGNFMLTQ) context deadline exceeded ======LOGS====== Jan 1 03:31:48.605861455 (636h2m48.4769445s) [upload=557331]: Starting upload Jan 1 03:31:48.605945095 (636h2m48.47702813s) [upload=557331]: Done starting upload Jan 1 03:31:48.606120018 (636h2m48.477203053s) [upload=557331]: Uploading segments... Jan 1 03:31:48.606250606 (636h2m48.477333641s) [upload=557331]: Got next segment. Inline: false Jan 1 03:31:48.606283599 (636h2m48.477366634s) [upload=557331]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:31:48.606288388 (636h2m48.477371423s) [upload=557331]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:31:48.851684150 (636h2m48.722767195s) [seg_pos={0 0}, upload=557331]: Begin upload segment... Jan 1 03:31:48.851741389 (636h2m48.722824434s) [seg_pos={0 0}, upload=557331]: Done begin upload segment. Jan 1 03:31:48.851777037 (636h2m48.722860082s) [noise=true , address=45.91.116.14:4192 , piece_id=HEAZI373ADYHLWCH64NG55TMGOC5QD2VJSFH5LIBEOVQAJSNQOXQ, seg_pos={0 0}, upload=557331]: Uploading piece... Jan 1 03:31:48.851882367 (636h2m48.722965412s) [noise=true , address=145.79.227.48:4303 , piece_id=YWOGX5KOTNIS272TFSWAI26S7NPT35622OYGU6DIHHFOZK7DSRIA, seg_pos={0 0}, upload=557331]: Uploading piece... Jan 1 03:31:48.852630432 (636h2m48.723713467s) [noise=true , address=45.91.116.11:4117 , piece_id=7OFTTAHEMU4YGOYH4AAVT57QPZFQ3GB2CIW6374MV3UOYGNFMLTQ, seg_pos={0 0}, upload=557331]: Uploading piece... Jan 1 03:31:48.852718369 (636h2m48.723801414s) [noise=true , address=45.91.116.9:4264 , piece_id=4UUZVQMLH6IX7V2GJ7J2W7TJLL3YL6CGM6TE7KSA47GKBJSFHCAA, seg_pos={0 0}, upload=557331]: Uploading piece... Jan 1 03:31:48.853214414 (636h2m48.724297459s) [noise=true , address=145.79.227.31:4799 , piece_id=YBIZK2CKC5J72G2UPTDOX6CFI6GYKACLBUJIVTWBCMNRCFESVIDA, seg_pos={0 0}, upload=557331]: Uploading piece... Jan 1 03:31:48.853786924 (636h2m48.724869959s) [noise=true , address=45.91.116.5:4298 , piece_id=33EEMPOVT3YIAVOI4Y5HDIERAPW23INCUKM7U7AJE4QUHPS4W5ZA, seg_pos={0 0}, upload=557331]: Uploading piece... Jan 1 03:31:48.854350647 (636h2m48.725433682s) [noise=true , address=145.79.227.46:4601 , piece_id=MJ4XDSFSFZWNFBR7ZNP6UMPK6XESIV6RAYAIQVOU4I6BEOBEWBKQ, seg_pos={0 0}, upload=557331]: Uploading piece... Jan 1 03:31:48.854424257 (636h2m48.725507292s) [noise=true , address=45.91.116.10:4082 , piece_id=HTYHF337UCIXD6IU7ARUB2OM7DE22D4PURZZHT26QX3KEJBYB63A, seg_pos={0 0}, upload=557331]: Uploading piece... Jan 1 03:31:48.855391830 (636h2m48.726474875s) [noise=true , address=145.79.227.6:4259 , piece_id=OVEKDISJ6JUL7CVF2XPRCZ27ILNFWOF22T6R7HV6N7HHRAYTN27Q, seg_pos={0 0}, upload=557331]: Uploading piece... Jan 1 03:31:48.855607961 (636h2m48.726691006s) [noise=true , address=145.79.227.22:4511 , piece_id=JRDU5A3LB4SXFZVFYB3HEIRCPD2X4YNKGM64UWGSL7APS3CBGWPA, seg_pos={0 0}, upload=557331]: Uploading piece... Jan 1 03:31:48.856570665 (636h2m48.7276537s) [noise=true , address=45.91.116.6:4974 , piece_id=4L2LSQNG773O6BDMRHRPVONEQT45NZH3V5V4WDPA5UB5RPA52C3Q, seg_pos={0 0}, upload=557331]: Uploading piece... Jan 1 03:31:48.857436083 (636h2m48.728519118s) [noise=true , address=145.79.227.15:4875 , piece_id=VNTJE3D43O6K4OIFMYMK4IQD76F7YLA7ZESWAYFQLHQU42B2AVFA, seg_pos={0 0}, upload=557331]: Uploading piece... Jan 1 03:31:48.858042507 (636h2m48.729125552s) [noise=true , address=145.79.227.51:4411 , piece_id=CD6RATWTKYQMK7H2QUPJXYED2GN5247OFJP4YDF5NKL3VNSNSWPA, seg_pos={0 0}, upload=557331]: Uploading piece... Jan 1 03:31:48.858702133 (636h2m48.729785178s) [noise=true , address=145.79.227.23:4547 , piece_id=6WQ4QYRTB3XKQ2N4CT7XXFAXXW34SLWAYHIGLQLPZEW2AKGGYZBA, seg_pos={0 0}, upload=557331]: Uploading piece... Jan 1 03:31:48.861019756 (636h2m48.732102791s) [noise=true , address=145.79.227.47:4266 , piece_id=S2IADG6ZKUOC2MCNGTDNLWKTGVLXJBSIZACZMMZG32SSQVWHYRMQ, seg_pos={0 0}, upload=557331]: Uploading piece... Jan 1 03:31:48.862034197 (636h2m48.733117232s) [noise=true , address=77.90.31.16:4735 , piece_id=ZQUXSBAAAMRTY5VH2T4RKPRHBHOJ74KBYAX2ROP53K6FT25SRG7Q, seg_pos={0 0}, upload=557331]: Uploading piece... Jan 1 03:31:50.450701976 (636h2m50.321785021s) [upload=557331]: Next returned nil segment Jan 1 03:31:50.450715532 (636h2m50.321798577s) [upload=557331]: Waiting for error group managing segments... Jan 1 03:31:50.832638738 (636h2m50.703721783s) [noise=true , address=145.79.227.23:4547 , piece_id=6WQ4QYRTB3XKQ2N4CT7XXFAXXW34SLWAYHIGLQLPZEW2AKGGYZBA, seg_pos={0 0}, upload=557331]: Done uploading piece. err: <nil> Jan 1 03:31:50.853744105 (636h2m50.72482715s) [noise=true , address=145.79.227.22:4511 , piece_id=JRDU5A3LB4SXFZVFYB3HEIRCPD2X4YNKGM64UWGSL7APS3CBGWPA, seg_pos={0 0}, upload=557331]: Done uploading piece. err: <nil> Jan 1 03:31:50.854304953 (636h2m50.725387999s) [noise=true , address=145.79.227.31:4799 , piece_id=YBIZK2CKC5J72G2UPTDOX6CFI6GYKACLBUJIVTWBCMNRCFESVIDA, seg_pos={0 0}, upload=557331]: Done uploading piece. err: <nil> Jan 1 03:31:50.858543283 (636h2m50.729626328s) [noise=true , address=145.79.227.6:4259 , piece_id=OVEKDISJ6JUL7CVF2XPRCZ27ILNFWOF22T6R7HV6N7HHRAYTN27Q, seg_pos={0 0}, upload=557331]: Done uploading piece. err: <nil> Jan 1 03:31:50.858656919 (636h2m50.729739964s) [noise=true , address=145.79.227.47:4266 , piece_id=S2IADG6ZKUOC2MCNGTDNLWKTGVLXJBSIZACZMMZG32SSQVWHYRMQ, seg_pos={0 0}, upload=557331]: Done uploading piece. err: <nil> Jan 1 03:31:50.865560475 (636h2m50.73664352s) [noise=true , address=145.79.227.48:4303 , piece_id=YWOGX5KOTNIS272TFSWAI26S7NPT35622OYGU6DIHHFOZK7DSRIA, seg_pos={0 0}, upload=557331]: Done uploading piece. err: <nil> Jan 1 03:31:50.928801002 (636h2m50.799884067s) [noise=true , address=145.79.227.15:4875 , piece_id=VNTJE3D43O6K4OIFMYMK4IQD76F7YLA7ZESWAYFQLHQU42B2AVFA, seg_pos={0 0}, upload=557331]: Done uploading piece. err: <nil> Jan 1 03:31:50.931695644 (636h2m50.802778689s) [noise=true , address=145.79.227.51:4411 , piece_id=CD6RATWTKYQMK7H2QUPJXYED2GN5247OFJP4YDF5NKL3VNSNSWPA, seg_pos={0 0}, upload=557331]: Done uploading piece. err: <nil> Jan 1 03:31:50.932461772 (636h2m50.803544807s) [noise=true , address=145.79.227.46:4601 , piece_id=MJ4XDSFSFZWNFBR7ZNP6UMPK6XESIV6RAYAIQVOU4I6BEOBEWBKQ, seg_pos={0 0}, upload=557331]: Done uploading piece. err: <nil> Jan 1 03:31:51.275350643 (636h2m51.146433688s) [noise=true , address=45.91.116.9:4264 , piece_id=4UUZVQMLH6IX7V2GJ7J2W7TJLL3YL6CGM6TE7KSA47GKBJSFHCAA, seg_pos={0 0}, upload=557331]: Done uploading piece. err: <nil> Jan 1 03:31:51.283898479 (636h2m51.154981524s) [noise=true , address=45.91.116.6:4974 , piece_id=4L2LSQNG773O6BDMRHRPVONEQT45NZH3V5V4WDPA5UB5RPA52C3Q, seg_pos={0 0}, upload=557331]: Done uploading piece. err: <nil> Jan 1 03:31:51.497298367 (636h2m51.368381402s) [noise=true , address=45.91.116.14:4192 , piece_id=HEAZI373ADYHLWCH64NG55TMGOC5QD2VJSFH5LIBEOVQAJSNQOXQ, seg_pos={0 0}, upload=557331]: Done uploading piece. err: <nil> Jan 1 03:31:51.526348722 (636h2m51.397431767s) [noise=true , address=45.91.116.11:4117 , piece_id=7OFTTAHEMU4YGOYH4AAVT57QPZFQ3GB2CIW6374MV3UOYGNFMLTQ, seg_pos={0 0}, upload=557331]: Done uploading piece. err: <nil> Jan 1 03:31:51.526375443 (636h2m51.397458488s) [seg_pos={0 0}, upload=557331]: Segment reached optimal threshold of 13 pieces. Jan 1 03:31:51.526610039 (636h2m51.397693074s) [noise=true , address=45.91.116.10:4082 , piece_id=HTYHF337UCIXD6IU7ARUB2OM7DE22D4PURZZHT26QX3KEJBYB63A, seg_pos={0 0}, upload=557331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Jan 1 03:31:51.526620600 (636h2m51.397703645s) [noise=true , address=45.91.116.10:4082 , piece_id=HTYHF337UCIXD6IU7ARUB2OM7DE22D4PURZZHT26QX3KEJBYB63A, seg_pos={0 0}, upload=557331]: Long tail cancellation due to optimal threshold Jan 1 03:31:51.526631481 (636h2m51.397714516s) [noise=true , address=77.90.31.16:4735 , piece_id=ZQUXSBAAAMRTY5VH2T4RKPRHBHOJ74KBYAX2ROP53K6FT25SRG7Q, seg_pos={0 0}, upload=557331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Jan 1 03:31:51.526636811 (636h2m51.397719856s) [noise=true , address=77.90.31.16:4735 , piece_id=ZQUXSBAAAMRTY5VH2T4RKPRHBHOJ74KBYAX2ROP53K6FT25SRG7Q, seg_pos={0 0}, upload=557331]: Long tail cancellation due to optimal threshold Jan 1 03:31:51.526769023 (636h2m51.397852068s) [noise=true , address=45.91.116.5:4298 , piece_id=33EEMPOVT3YIAVOI4Y5HDIERAPW23INCUKM7U7AJE4QUHPS4W5ZA, seg_pos={0 0}, upload=557331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:49332->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:49332->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49332->45.91.116.5:4298: use of closed network connection Jan 1 03:31:51.526792457 (636h2m51.397875493s) [noise=true , address=45.91.116.5:4298 , piece_id=33EEMPOVT3YIAVOI4Y5HDIERAPW23INCUKM7U7AJE4QUHPS4W5ZA, seg_pos={0 0}, upload=557331]: Long tail cancellation due to optimal threshold Jan 1 03:31:51.526810060 (636h2m51.397893096s) [seg_pos={0 0}, upload=557331]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:31:51.526945388 (636h2m51.398028423s) [upload=557331]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:31:51.526975907 (636h2m51.398058952s) [upload=557331]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:31:51.813606695 (636h2m51.68468973s) [upload=557331]: Done uploading segments... Jan 1 03:31:51.813655197 (636h2m51.684738242s) [upload=557331]: Upload finished. err: <nil> Duration: 30.003895842s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X....X.................................] duration: 15.604231932s trace: https://jaeger.dev.storj.io/trace/725fc16e891a4127 FAIL (2026-01-01 03:32:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3:03:23.412681369 (635h34m23.283764404s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413214023 (635h34m23.284297058s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413504245 (635h34m23.28458728s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413821750 (635h34m23.284904785s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414124316 (635h34m23.285207351s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414417554 (635h34m23.285500589s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414689172 (635h34m23.285772207s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415023338 (635h34m23.286106373s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415341384 (635h34m23.286424419s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415659309 (635h34m23.286742354s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416184289 (635h34m23.287267324s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416571025 (635h34m23.28765406s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416998890 (635h34m23.288081925s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.417376770 (635h34m23.288459815s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.418791664 (635h34m23.289874709s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.420183384 (635h34m23.291266419s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.421563200 (635h34m23.292646246s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:24.823751878 (635h34m24.694834913s) [upload=556915]: Next returned nil segment Jan 1 03:03:24.823765594 (635h34m24.694848629s) [upload=556915]: Waiting for error group managing segments... Jan 1 03:03:25.191603537 (635h34m25.062686582s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.271741648 (635h34m25.142824683s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.278781112 (635h34m25.149864157s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.337014705 (635h34m25.20809774s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.372659678 (635h34m25.243742713s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.439670894 (635h34m25.31075394s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.447380494 (635h34m25.318463529s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.459093186 (635h34m25.330176231s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.602592395 (635h34m25.47367543s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.719366504 (635h34m25.59044954s) [noise=true , address=45.91.116.12:4230 , piece_id=WQHO3KOUTZSHBZIDUXANBVFSUKTBKEWHTZ3Z7BNGXGOFP2BMJHWA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.875237800 (635h34m25.746320845s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.143929177 (635h34m26.015012212s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.159675169 (635h34m26.030758204s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181352455 (635h34m26.05243549s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181379146 (635h34m26.052462181s) [seg_pos={0 0}, upload=556915]: Segment reached optimal threshold of 14 pieces. Jan 1 03:03:26.181716378 (635h34m26.052799413s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Pvm1SftEF3tnCEBqUn1ZmyVcLDBkR6pyegEb7mtzjdQiiqYqt): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181723281 (635h34m26.052806316s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181784638 (635h34m26.052867703s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181848530 (635h34m26.052931565s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181859921 (635h34m26.052942956s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181866474 (635h34m26.052949509s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181920486 (635h34m26.053003522s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:184g6qHdezLW3d2CKNW8GRgdzeVLQuMEhd8B8c1zEnkKNkeSTu): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181925296 (635h34m26.053008331s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181941377 (635h34m26.053024412s) [seg_pos={0 0}, upload=556915]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 03:03:26.182003695 (635h34m26.05308673s) [upload=556915]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:03:26.182009416 (635h34m26.053092461s) [upload=556915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:03:26.474026257 (635h34m26.345109292s) [upload=556915]: Done uploading segments... Jan 1 03:03:26.474049391 (635h34m26.345132427s) [upload=556915]: Upload finished. err: <nil> Duration: 15.604264863s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................X......X............................] duration: 3.453195856s trace: https://jaeger.dev.storj.io/trace/679257cbe75708e0 FAIL (2026-01-01 03:31:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.453219551s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X..X.XX.......X...X..X..X...X..X....XXX.X.X..XX.X.XXX..XX..X] duration: 30.003947624s trace: https://jaeger.dev.storj.io/trace/1550ddbbab5b6c39 FAIL (2026-01-01 03:31:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 1 03:31:07.469686587 (636h2m7.340769632s) [upload=557323]: Starting upload Jan 1 03:31:07.469790154 (636h2m7.340873209s) [upload=557323]: Done starting upload Jan 1 03:31:07.470264507 (636h2m7.341347563s) [upload=557323]: Uploading segments... Jan 1 03:31:07.470524642 (636h2m7.341607687s) [upload=557323]: Got next segment. Inline: false Jan 1 03:31:07.470589897 (636h2m7.341672942s) [upload=557323]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:31:07.470598633 (636h2m7.341681668s) [upload=557323]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:31:07.716236927 (636h2m7.587319983s) [seg_pos={0 0}, upload=557323]: Begin upload segment... Jan 1 03:31:07.716285550 (636h2m7.587368585s) [seg_pos={0 0}, upload=557323]: Done begin upload segment. Jan 1 03:31:07.716313022 (636h2m7.587396057s) [noise=true , address=145.79.227.47:4265 , piece_id=WB55XDV35I46SMXDJPNHVR75CKVBGN2IPCXRJ5BW43PAREGMOY5A, seg_pos={0 0}, upload=557323]: Uploading piece... Jan 1 03:31:07.716332039 (636h2m7.587415084s) [noise=true , address=77.90.31.14:4662 , piece_id=3GANCUQNZXRAJA26TPECB27Z62KXFSY2AUHCGKDI7GXPKWBN4NZA, seg_pos={0 0}, upload=557323]: Uploading piece... Jan 1 03:31:07.717045807 (636h2m7.588128843s) [noise=true , address=145.79.227.14:4839 , piece_id=IJZULMS2SPTUEI7PHIALYIFUSEVN7Q6BVZYYROPPJUWWHZCM4WIQ, seg_pos={0 0}, upload=557323]: Uploading piece... Jan 1 03:31:07.717276156 (636h2m7.588359202s) [noise=true , address=145.79.227.11:4730 , piece_id=NVVIH7QKUA7MY7VJZ7ACX4VCRQRJAQDPDJZ2FPSXM4X6SMZ5U2EQ, seg_pos={0 0}, upload=557323]: Uploading piece... Jan 1 03:31:07.717411213 (636h2m7.588494248s) [noise=true , address=145.79.227.36:4518 , piece_id=4GL5DYGFAB7SIEKE4XCFZDAHERPCVJXYO2BWYXHHUCX5KF27MEZA, seg_pos={0 0}, upload=557323]: Uploading piece... Jan 1 03:31:07.717686327 (636h2m7.588769382s) [noise=true , address=45.91.116.10:4084 , piece_id=6CGRTZQ2GST3W77PSP6QNUEQK6H63K65GFTUSNGD5LZ2C3CJNZ7Q, seg_pos={0 0}, upload=557323]: Uploading piece... Jan 1 03:31:07.717774345 (636h2m7.58885739s) [noise=true , address=45.91.116.5:4297 , piece_id=NBB3P6HJG2HFX47NYZYACBTXWEHHXYRK24UVBNR4VEWYQ343ZJ3A, seg_pos={0 0}, upload=557323]: Uploading piece... Jan 1 03:31:07.718179816 (636h2m7.589262851s) [noise=true , address=45.91.116.13:4153 , piece_id=IHPQAPEWNFBDBFUSCV5IG265JM3TT3OS4A4GSJKPCEUGYHB3XQEA, seg_pos={0 0}, upload=557323]: Uploading piece... Jan 1 03:31:07.718277923 (636h2m7.589360968s) [noise=true , address=145.79.227.9:4367 , piece_id=KFO2VO2JA7P6V76LK3XEGKLOBCLDL2U2SZUDVTTP3Y7ITPCT77OA, seg_pos={0 0}, upload=557323]: Uploading piece... Jan 1 03:31:07.718540373 (636h2m7.589623419s) [noise=true , address=45.91.116.12:4225 , piece_id=2BBQK35HZLTSEU5G3OJGBONRIMK2BIIB4PJ7NBNXFHBBX5OO5V4A, seg_pos={0 0}, upload=557323]: Uploading piece... Jan 1 03:31:07.718766143 (636h2m7.589849188s) [noise=true , address=145.79.227.37:4567 , piece_id=FJME3UD4PLWB2I7Y2QBJRD3T3HTWOIRQF33D5BALC7RXJGHTKUOQ, seg_pos={0 0}, upload=557323]: Uploading piece... Jan 1 03:31:07.718854992 (636h2m7.589938028s) [noise=true , address=45.91.116.8:4047 , piece_id=JFTUNIQBFGPHULLYXDFKQNKFALVMZZX5II3OLK2GG3XUYXADYY3A, seg_pos={0 0}, upload=557323]: Uploading piece... Jan 1 03:31:07.719231099 (636h2m7.590314164s) [noise=true , address=77.90.31.10:4519 , piece_id=JXMMGHDTE2DWICVKE7XJXD7KPIHRMZUQHHGE3WAX5EJM77FILJLA, seg_pos={0 0}, upload=557323]: Uploading piece... Jan 1 03:31:07.719261436 (636h2m7.590344481s) [noise=true , address=145.79.227.46:4601 , piece_id=LEWAZZCB6OZWFF42ORNKD5J64XCM77535UN6D7HJCUVPGNCGYNKQ, seg_pos={0 0}, upload=557323]: Uploading piece... Jan 1 03:31:07.719624768 (636h2m7.590707813s) [noise=true , address=45.91.116.14:4190 , piece_id=2LCFPMD44GTMBDP2K5WG3HZTSCMFV2PKUK5RK2XLHZHXTJLAR6EQ, seg_pos={0 0}, upload=557323]: Uploading piece... Jan 1 03:31:07.719778010 (636h2m7.590861055s) [noise=true , address=145.79.227.35:4482 , piece_id=7LE3AVT25DWI44OY6YOVVA2TKNTYLJOI42UTIEQZXUVRCSZXIY5Q, seg_pos={0 0}, upload=557323]: Uploading piece... Jan 1 03:31:09.324293023 (636h2m9.195376068s) [upload=557323]: Next returned nil segment Jan 1 03:31:09.324308522 (636h2m9.195391557s) [upload=557323]: Waiting for error group managing segments... Jan 1 03:31:09.706039543 (636h2m9.577122588s) [noise=true , address=145.79.227.14:4839 , piece_id=IJZULMS2SPTUEI7PHIALYIFUSEVN7Q6BVZYYROPPJUWWHZCM4WIQ, seg_pos={0 0}, upload=557323]: Done uploading piece. err: <nil> Jan 1 03:31:09.711336018 (636h2m9.582419063s) [noise=true , address=145.79.227.47:4265 , piece_id=WB55XDV35I46SMXDJPNHVR75CKVBGN2IPCXRJ5BW43PAREGMOY5A, seg_pos={0 0}, upload=557323]: Done uploading piece. err: <nil> Jan 1 03:31:09.723698397 (636h2m9.594781432s) [noise=true , address=145.79.227.46:4601 , piece_id=LEWAZZCB6OZWFF42ORNKD5J64XCM77535UN6D7HJCUVPGNCGYNKQ, seg_pos={0 0}, upload=557323]: Done uploading piece. err: <nil> Jan 1 03:31:09.733620260 (636h2m9.604703325s) [noise=true , address=145.79.227.37:4567 , piece_id=FJME3UD4PLWB2I7Y2QBJRD3T3HTWOIRQF33D5BALC7RXJGHTKUOQ, seg_pos={0 0}, upload=557323]: Done uploading piece. err: <nil> Jan 1 03:31:09.733926272 (636h2m9.605009317s) [noise=true , address=145.79.227.9:4367 , piece_id=KFO2VO2JA7P6V76LK3XEGKLOBCLDL2U2SZUDVTTP3Y7ITPCT77OA, seg_pos={0 0}, upload=557323]: Done uploading piece. err: <nil> Jan 1 03:31:09.789353011 (636h2m9.660436056s) [noise=true , address=145.79.227.11:4730 , piece_id=NVVIH7QKUA7MY7VJZ7ACX4VCRQRJAQDPDJZ2FPSXM4X6SMZ5U2EQ, seg_pos={0 0}, upload=557323]: Done uploading piece. err: <nil> Jan 1 03:31:09.792729741 (636h2m9.663812786s) [noise=true , address=145.79.227.36:4518 , piece_id=4GL5DYGFAB7SIEKE4XCFZDAHERPCVJXYO2BWYXHHUCX5KF27MEZA, seg_pos={0 0}, upload=557323]: Done uploading piece. err: <nil> Jan 1 03:31:09.915070151 (636h2m9.786153196s) [noise=true , address=77.90.31.14:4662 , piece_id=3GANCUQNZXRAJA26TPECB27Z62KXFSY2AUHCGKDI7GXPKWBN4NZA, seg_pos={0 0}, upload=557323]: Done uploading piece. err: <nil> Jan 1 03:31:10.045987484 (636h2m9.917070519s) [noise=true , address=45.91.116.5:4297 , piece_id=NBB3P6HJG2HFX47NYZYACBTXWEHHXYRK24UVBNR4VEWYQ343ZJ3A, seg_pos={0 0}, upload=557323]: Done uploading piece. err: <nil> Jan 1 03:31:10.125110994 (636h2m9.996194029s) [noise=true , address=77.90.31.10:4519 , piece_id=JXMMGHDTE2DWICVKE7XJXD7KPIHRMZUQHHGE3WAX5EJM77FILJLA, seg_pos={0 0}, upload=557323]: Done uploading piece. err: <nil> Jan 1 03:31:10.160674340 (636h2m10.031757385s) [noise=true , address=45.91.116.12:4225 , piece_id=2BBQK35HZLTSEU5G3OJGBONRIMK2BIIB4PJ7NBNXFHBBX5OO5V4A, seg_pos={0 0}, upload=557323]: Done uploading piece. err: <nil> Jan 1 03:31:10.435284059 (636h2m10.306367105s) [noise=true , address=45.91.116.10:4084 , piece_id=6CGRTZQ2GST3W77PSP6QNUEQK6H63K65GFTUSNGD5LZ2C3CJNZ7Q, seg_pos={0 0}, upload=557323]: Done uploading piece. err: <nil> Jan 1 03:31:10.948147040 (636h2m10.819230085s) [noise=true , address=45.91.116.8:4047 , piece_id=JFTUNIQBFGPHULLYXDFKQNKFALVMZZX5II3OLK2GG3XUYXADYY3A, seg_pos={0 0}, upload=557323]: Done uploading piece. err: <nil> Jan 1 03:31:10.948172018 (636h2m10.819255063s) [seg_pos={0 0}, upload=557323]: Segment reached optimal threshold of 13 pieces. Jan 1 03:31:10.948406936 (636h2m10.819489981s) [noise=true , address=45.91.116.13:4153 , piece_id=IHPQAPEWNFBDBFUSCV5IG265JM3TT3OS4A4GSJKPCEUGYHB3XQEA, seg_pos={0 0}, upload=557323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:49244->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:49244->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49244->45.91.116.13:4153: use of closed network connection Jan 1 03:31:10.948415462 (636h2m10.819498507s) [noise=true , address=145.79.227.35:4482 , piece_id=7LE3AVT25DWI44OY6YOVVA2TKNTYLJOI42UTIEQZXUVRCSZXIY5Q, seg_pos={0 0}, upload=557323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Jan 1 03:31:10.948429869 (636h2m10.819512924s) [noise=true , address=145.79.227.35:4482 , piece_id=7LE3AVT25DWI44OY6YOVVA2TKNTYLJOI42UTIEQZXUVRCSZXIY5Q, seg_pos={0 0}, upload=557323]: Long tail cancellation due to optimal threshold Jan 1 03:31:10.948483171 (636h2m10.819566216s) [noise=true , address=45.91.116.13:4153 , piece_id=IHPQAPEWNFBDBFUSCV5IG265JM3TT3OS4A4GSJKPCEUGYHB3XQEA, seg_pos={0 0}, upload=557323]: Long tail cancellation due to optimal threshold Jan 1 03:31:10.948591126 (636h2m10.819674171s) [noise=true , address=45.91.116.14:4190 , piece_id=2LCFPMD44GTMBDP2K5WG3HZTSCMFV2PKUK5RK2XLHZHXTJLAR6EQ, seg_pos={0 0}, upload=557323]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Jan 1 03:31:10.948599852 (636h2m10.819682898s) [noise=true , address=45.91.116.14:4190 , piece_id=2LCFPMD44GTMBDP2K5WG3HZTSCMFV2PKUK5RK2XLHZHXTJLAR6EQ, seg_pos={0 0}, upload=557323]: Long tail cancellation due to optimal threshold Jan 1 03:31:10.948618999 (636h2m10.819702044s) [seg_pos={0 0}, upload=557323]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:31:10.948728558 (636h2m10.819811603s) [upload=557323]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:31:10.948735952 (636h2m10.819818997s) [upload=557323]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:31:11.244650798 (636h2m11.115733843s) [upload=557323]: Done uploading segments... Jan 1 03:31:11.244676417 (636h2m11.115759462s) [upload=557323]: Upload finished. err: <nil> Duration: 30.003969919s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X....X................................] duration: 37.225324937s trace: https://jaeger.dev.storj.io/trace/725fc16e891a4127 FAIL (2026-01-01 03:31:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3:03:23.412681369 (635h34m23.283764404s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413214023 (635h34m23.284297058s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413504245 (635h34m23.28458728s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413821750 (635h34m23.284904785s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414124316 (635h34m23.285207351s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414417554 (635h34m23.285500589s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414689172 (635h34m23.285772207s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415023338 (635h34m23.286106373s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415341384 (635h34m23.286424419s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415659309 (635h34m23.286742354s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416184289 (635h34m23.287267324s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416571025 (635h34m23.28765406s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416998890 (635h34m23.288081925s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.417376770 (635h34m23.288459815s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.418791664 (635h34m23.289874709s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.420183384 (635h34m23.291266419s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.421563200 (635h34m23.292646246s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:24.823751878 (635h34m24.694834913s) [upload=556915]: Next returned nil segment Jan 1 03:03:24.823765594 (635h34m24.694848629s) [upload=556915]: Waiting for error group managing segments... Jan 1 03:03:25.191603537 (635h34m25.062686582s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.271741648 (635h34m25.142824683s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.278781112 (635h34m25.149864157s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.337014705 (635h34m25.20809774s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.372659678 (635h34m25.243742713s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.439670894 (635h34m25.31075394s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.447380494 (635h34m25.318463529s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.459093186 (635h34m25.330176231s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.602592395 (635h34m25.47367543s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.719366504 (635h34m25.59044954s) [noise=true , address=45.91.116.12:4230 , piece_id=WQHO3KOUTZSHBZIDUXANBVFSUKTBKEWHTZ3Z7BNGXGOFP2BMJHWA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.875237800 (635h34m25.746320845s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.143929177 (635h34m26.015012212s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.159675169 (635h34m26.030758204s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181352455 (635h34m26.05243549s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181379146 (635h34m26.052462181s) [seg_pos={0 0}, upload=556915]: Segment reached optimal threshold of 14 pieces. Jan 1 03:03:26.181716378 (635h34m26.052799413s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Pvm1SftEF3tnCEBqUn1ZmyVcLDBkR6pyegEb7mtzjdQiiqYqt): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181723281 (635h34m26.052806316s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181784638 (635h34m26.052867703s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181848530 (635h34m26.052931565s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181859921 (635h34m26.052942956s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181866474 (635h34m26.052949509s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181920486 (635h34m26.053003522s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:184g6qHdezLW3d2CKNW8GRgdzeVLQuMEhd8B8c1zEnkKNkeSTu): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181925296 (635h34m26.053008331s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181941377 (635h34m26.053024412s) [seg_pos={0 0}, upload=556915]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 03:03:26.182003695 (635h34m26.05308673s) [upload=556915]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:03:26.182009416 (635h34m26.053092461s) [upload=556915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:03:26.474026257 (635h34m26.345109292s) [upload=556915]: Done uploading segments... Jan 1 03:03:26.474049391 (635h34m26.345132427s) [upload=556915]: Upload finished. err: <nil> Duration: 37.225342309s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X....X...............................] duration: 40.552983961s trace: https://jaeger.dev.storj.io/trace/725fc16e891a4127 FAIL (2026-01-01 03:30:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3:03:23.412681369 (635h34m23.283764404s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413214023 (635h34m23.284297058s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413504245 (635h34m23.28458728s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413821750 (635h34m23.284904785s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414124316 (635h34m23.285207351s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414417554 (635h34m23.285500589s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414689172 (635h34m23.285772207s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415023338 (635h34m23.286106373s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415341384 (635h34m23.286424419s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415659309 (635h34m23.286742354s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416184289 (635h34m23.287267324s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416571025 (635h34m23.28765406s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416998890 (635h34m23.288081925s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.417376770 (635h34m23.288459815s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.418791664 (635h34m23.289874709s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.420183384 (635h34m23.291266419s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.421563200 (635h34m23.292646246s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:24.823751878 (635h34m24.694834913s) [upload=556915]: Next returned nil segment Jan 1 03:03:24.823765594 (635h34m24.694848629s) [upload=556915]: Waiting for error group managing segments... Jan 1 03:03:25.191603537 (635h34m25.062686582s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.271741648 (635h34m25.142824683s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.278781112 (635h34m25.149864157s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.337014705 (635h34m25.20809774s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.372659678 (635h34m25.243742713s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.439670894 (635h34m25.31075394s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.447380494 (635h34m25.318463529s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.459093186 (635h34m25.330176231s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.602592395 (635h34m25.47367543s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.719366504 (635h34m25.59044954s) [noise=true , address=45.91.116.12:4230 , piece_id=WQHO3KOUTZSHBZIDUXANBVFSUKTBKEWHTZ3Z7BNGXGOFP2BMJHWA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.875237800 (635h34m25.746320845s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.143929177 (635h34m26.015012212s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.159675169 (635h34m26.030758204s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181352455 (635h34m26.05243549s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181379146 (635h34m26.052462181s) [seg_pos={0 0}, upload=556915]: Segment reached optimal threshold of 14 pieces. Jan 1 03:03:26.181716378 (635h34m26.052799413s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Pvm1SftEF3tnCEBqUn1ZmyVcLDBkR6pyegEb7mtzjdQiiqYqt): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181723281 (635h34m26.052806316s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181784638 (635h34m26.052867703s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181848530 (635h34m26.052931565s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181859921 (635h34m26.052942956s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181866474 (635h34m26.052949509s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181920486 (635h34m26.053003522s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:184g6qHdezLW3d2CKNW8GRgdzeVLQuMEhd8B8c1zEnkKNkeSTu): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181925296 (635h34m26.053008331s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181941377 (635h34m26.053024412s) [seg_pos={0 0}, upload=556915]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 03:03:26.182003695 (635h34m26.05308673s) [upload=556915]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:03:26.182009416 (635h34m26.053092461s) [upload=556915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:03:26.474026257 (635h34m26.345109292s) [upload=556915]: Done uploading segments... Jan 1 03:03:26.474049391 (635h34m26.345132427s) [upload=556915]: Upload finished. err: <nil> Duration: 40.553015251s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................X......X...........................] duration: 3.582600724s trace: https://jaeger.dev.storj.io/trace/679257cbe75708e0 FAIL (2026-01-01 03:30:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.583199685s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XX..X.XX.......X...X..X..X...X..X....XXX.X.X..XX.X.XXX..XX..] duration: 6.16969767s trace: https://jaeger.dev.storj.io/trace/579312301588aa45 FAIL (2026-01-01 03:30:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 03:28:37.501792999 (635h59m37.372876034s) [upload=557288]: Starting upload Jan 1 03:28:37.501873622 (635h59m37.372956658s) [upload=557288]: Done starting upload Jan 1 03:28:37.501912718 (635h59m37.372995753s) [upload=557288]: Uploading segments... Jan 1 03:28:37.502049948 (635h59m37.373132983s) [upload=557288]: Got next segment. Inline: false Jan 1 03:28:37.502089003 (635h59m37.373172048s) [upload=557288]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:28:37.502092750 (635h59m37.373175785s) [upload=557288]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:28:37.752742837 (635h59m37.623825872s) [seg_pos={0 0}, upload=557288]: Begin upload segment... Jan 1 03:28:37.752797551 (635h59m37.623880586s) [seg_pos={0 0}, upload=557288]: Done begin upload segment. Jan 1 03:28:37.752848899 (635h59m37.623931934s) [noise=true , address=145.79.227.51:4409 , piece_id=SQ4EOSQSDNDNKSEHDII3THHDARC6KFUP6CIWLR6OSTHMNM6HOBRQ, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.752863076 (635h59m37.623946121s) [noise=true , address=145.79.227.42:4122 , piece_id=JR3I4TEANOK55JT6WUSMZFMYZANF3W4RHVNQZLSFGWB4UJW5EIEA, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.753027038 (635h59m37.624110073s) [noise=true , address=45.91.116.10:4082 , piece_id=CSU66KWPTC2T2GBSIKGVC6FGYN5CFMDMCL5GECFRKHZKUPLE344Q, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.753381403 (635h59m37.624464438s) [noise=true , address=145.79.227.12:4767 , piece_id=TKMTCS2HQIUYNPDBK37P3CO7WZCIRHXWOHWCTCMBXSLPTF3HYT7A, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.753719266 (635h59m37.624802301s) [noise=true , address=145.79.227.11:4731 , piece_id=AB6E35YEK2XXETABVYVZDMAQ7X2MSNPMZVS4HLRCJN3U7C23DDXA, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.754024697 (635h59m37.625107742s) [noise=true , address=145.79.227.6:4259 , piece_id=Q2SUUS4KLKVTQOR5B7JDACBDNC2OZKHM5XH2HP524JJC4UV4CZ2Q, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.754282388 (635h59m37.625365434s) [noise=true , address=45.91.116.6:4974 , piece_id=QH7FF4YUFECLC5Y6K6HJBNNBXFANOEEOZBCCAGAA7V2SE3X6PKLQ, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.754547863 (635h59m37.625630898s) [noise=true , address=77.90.31.19:4842 , piece_id=H6BNUQH7VX5WPSQZFEQJDKWHKCJRATK5EYQTZFVUFLVEYFLQMAFQ, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.754874175 (635h59m37.62595722s) [noise=true , address=45.91.116.5:4297 , piece_id=465KFYYTC3S6RVBFWRUGRUPEMDYKYMFK4V4Y7NLOQIR37P7JAFVA, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.755242065 (635h59m37.6263251s) [noise=true , address=145.79.227.8:4329 , piece_id=ZULYGSQMCGB62PBUBWF5FLXGP6UAQV2ZT4MSXR3VWATIQQWKOCUQ, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.756494821 (635h59m37.627577856s) [noise=true , address=45.91.116.8:4047 , piece_id=4RTHEAZYI4HQSMP2YGM4RSZXAZREO27VVLPY7VRGVHSBCY3PURBQ, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.758165121 (635h59m37.629248156s) [noise=true , address=145.79.227.2:4113 , piece_id=KUVJGHYQ4YR7LAFPOTINTI2PK42ANFPSGHRFRBXVOPQFRW36XZGQ, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.759195824 (635h59m37.630278859s) [noise=true , address=145.79.227.31:4798 , piece_id=M7PPLKAVUD7UJDN5SBRIEQMUPEQ3TCCRZE7OJ7CMRBFFGWDNS4FQ, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.760476342 (635h59m37.631559387s) [noise=true , address=145.79.227.45:4231 , piece_id=CDVCVV3WXHL3NFCAALHSPP5U3KLEUSSVECS65VLGEAEQNMNECY3A, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.762002508 (635h59m37.633085543s) [noise=true , address=77.90.31.10:4517 , piece_id=UMJRZGBF2XLCXBHQLR2WJDUHW5BW25VZIOFFSQMGADHCJGNFY7LQ, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.763280322 (635h59m37.634363357s) [noise=true , address=145.79.227.35:4481 , piece_id=2ZQHNAAWJRJD3IBRIGWYL5TCM5I6ATPDKUWV6ND7ARMFVI4PTVNQ, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:39.457289841 (635h59m39.328372876s) [upload=557288]: Next returned nil segment Jan 1 03:28:39.457302396 (635h59m39.328385441s) [upload=557288]: Waiting for error group managing segments... Jan 1 03:28:39.730758959 (635h59m39.601842004s) [noise=true , address=145.79.227.11:4731 , piece_id=AB6E35YEK2XXETABVYVZDMAQ7X2MSNPMZVS4HLRCJN3U7C23DDXA, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:39.747123368 (635h59m39.618206413s) [noise=true , address=145.79.227.51:4409 , piece_id=SQ4EOSQSDNDNKSEHDII3THHDARC6KFUP6CIWLR6OSTHMNM6HOBRQ, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:39.763070162 (635h59m39.634153258s) [noise=true , address=145.79.227.6:4259 , piece_id=Q2SUUS4KLKVTQOR5B7JDACBDNC2OZKHM5XH2HP524JJC4UV4CZ2Q, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:39.765792416 (635h59m39.636875451s) [noise=true , address=145.79.227.42:4122 , piece_id=JR3I4TEANOK55JT6WUSMZFMYZANF3W4RHVNQZLSFGWB4UJW5EIEA, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:39.769098060 (635h59m39.640181106s) [noise=true , address=145.79.227.8:4329 , piece_id=ZULYGSQMCGB62PBUBWF5FLXGP6UAQV2ZT4MSXR3VWATIQQWKOCUQ, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:39.769254900 (635h59m39.640337935s) [noise=true , address=145.79.227.12:4767 , piece_id=TKMTCS2HQIUYNPDBK37P3CO7WZCIRHXWOHWCTCMBXSLPTF3HYT7A, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:39.772290961 (635h59m39.643373996s) [noise=true , address=145.79.227.2:4113 , piece_id=KUVJGHYQ4YR7LAFPOTINTI2PK42ANFPSGHRFRBXVOPQFRW36XZGQ, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:39.774741276 (635h59m39.645824311s) [noise=true , address=145.79.227.31:4798 , piece_id=M7PPLKAVUD7UJDN5SBRIEQMUPEQ3TCCRZE7OJ7CMRBFFGWDNS4FQ, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:39.780832684 (635h59m39.651915729s) [noise=true , address=145.79.227.45:4231 , piece_id=CDVCVV3WXHL3NFCAALHSPP5U3KLEUSSVECS65VLGEAEQNMNECY3A, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:39.908369770 (635h59m39.779452805s) [noise=true , address=77.90.31.19:4842 , piece_id=H6BNUQH7VX5WPSQZFEQJDKWHKCJRATK5EYQTZFVUFLVEYFLQMAFQ, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:40.081380001 (635h59m39.952463036s) [noise=true , address=45.91.116.6:4974 , piece_id=QH7FF4YUFECLC5Y6K6HJBNNBXFANOEEOZBCCAGAA7V2SE3X6PKLQ, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:40.258939746 (635h59m40.130022791s) [noise=true , address=45.91.116.5:4297 , piece_id=465KFYYTC3S6RVBFWRUGRUPEMDYKYMFK4V4Y7NLOQIR37P7JAFVA, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:40.636022335 (635h59m40.50710537s) [noise=true , address=77.90.31.10:4517 , piece_id=UMJRZGBF2XLCXBHQLR2WJDUHW5BW25VZIOFFSQMGADHCJGNFY7LQ, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:40.636048265 (635h59m40.5071313s) [seg_pos={0 0}, upload=557288]: Segment reached optimal threshold of 13 pieces. Jan 1 03:28:40.636326595 (635h59m40.50740963s) [noise=true , address=45.91.116.10:4082 , piece_id=CSU66KWPTC2T2GBSIKGVC6FGYN5CFMDMCL5GECFRKHZKUPLE344Q, seg_pos={0 0}, upload=557288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:41672->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:41672->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41672->45.91.116.10:4082: use of closed network connection Jan 1 03:28:40.636337966 (635h59m40.507421001s) [noise=true , address=45.91.116.10:4082 , piece_id=CSU66KWPTC2T2GBSIKGVC6FGYN5CFMDMCL5GECFRKHZKUPLE344Q, seg_pos={0 0}, upload=557288]: Long tail cancellation due to optimal threshold Jan 1 03:28:40.636511677 (635h59m40.507594712s) [noise=true , address=45.91.116.8:4047 , piece_id=4RTHEAZYI4HQSMP2YGM4RSZXAZREO27VVLPY7VRGVHSBCY3PURBQ, seg_pos={0 0}, upload=557288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Jan 1 03:28:40.636533639 (635h59m40.507616674s) [noise=true , address=45.91.116.8:4047 , piece_id=4RTHEAZYI4HQSMP2YGM4RSZXAZREO27VVLPY7VRGVHSBCY3PURBQ, seg_pos={0 0}, upload=557288]: Long tail cancellation due to optimal threshold Jan 1 03:28:40.636557795 (635h59m40.50764083s) [noise=true , address=145.79.227.35:4481 , piece_id=2ZQHNAAWJRJD3IBRIGWYL5TCM5I6ATPDKUWV6ND7ARMFVI4PTVNQ, seg_pos={0 0}, upload=557288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Jan 1 03:28:40.636586881 (635h59m40.507669916s) [noise=true , address=145.79.227.35:4481 , piece_id=2ZQHNAAWJRJD3IBRIGWYL5TCM5I6ATPDKUWV6ND7ARMFVI4PTVNQ, seg_pos={0 0}, upload=557288]: Long tail cancellation due to optimal threshold Jan 1 03:28:40.636609183 (635h59m40.507692228s) [seg_pos={0 0}, upload=557288]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:28:40.636721086 (635h59m40.507804131s) [upload=557288]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:28:40.636766863 (635h59m40.507849908s) [upload=557288]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:28:40.929867821 (635h59m40.800950856s) [upload=557288]: Done uploading segments... Jan 1 03:28:40.929886277 (635h59m40.800969312s) [upload=557288]: Upload finished. err: <nil> Duration: 6.169736733s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X....X..............................] duration: 57.3211933s trace: https://jaeger.dev.storj.io/trace/725fc16e891a4127 FAIL (2026-01-01 03:30:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3:03:23.412681369 (635h34m23.283764404s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413214023 (635h34m23.284297058s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413504245 (635h34m23.28458728s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413821750 (635h34m23.284904785s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414124316 (635h34m23.285207351s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414417554 (635h34m23.285500589s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414689172 (635h34m23.285772207s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415023338 (635h34m23.286106373s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415341384 (635h34m23.286424419s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415659309 (635h34m23.286742354s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416184289 (635h34m23.287267324s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416571025 (635h34m23.28765406s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416998890 (635h34m23.288081925s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.417376770 (635h34m23.288459815s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.418791664 (635h34m23.289874709s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.420183384 (635h34m23.291266419s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.421563200 (635h34m23.292646246s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:24.823751878 (635h34m24.694834913s) [upload=556915]: Next returned nil segment Jan 1 03:03:24.823765594 (635h34m24.694848629s) [upload=556915]: Waiting for error group managing segments... Jan 1 03:03:25.191603537 (635h34m25.062686582s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.271741648 (635h34m25.142824683s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.278781112 (635h34m25.149864157s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.337014705 (635h34m25.20809774s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.372659678 (635h34m25.243742713s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.439670894 (635h34m25.31075394s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.447380494 (635h34m25.318463529s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.459093186 (635h34m25.330176231s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.602592395 (635h34m25.47367543s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.719366504 (635h34m25.59044954s) [noise=true , address=45.91.116.12:4230 , piece_id=WQHO3KOUTZSHBZIDUXANBVFSUKTBKEWHTZ3Z7BNGXGOFP2BMJHWA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.875237800 (635h34m25.746320845s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.143929177 (635h34m26.015012212s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.159675169 (635h34m26.030758204s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181352455 (635h34m26.05243549s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181379146 (635h34m26.052462181s) [seg_pos={0 0}, upload=556915]: Segment reached optimal threshold of 14 pieces. Jan 1 03:03:26.181716378 (635h34m26.052799413s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Pvm1SftEF3tnCEBqUn1ZmyVcLDBkR6pyegEb7mtzjdQiiqYqt): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181723281 (635h34m26.052806316s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181784638 (635h34m26.052867703s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181848530 (635h34m26.052931565s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181859921 (635h34m26.052942956s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181866474 (635h34m26.052949509s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181920486 (635h34m26.053003522s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:184g6qHdezLW3d2CKNW8GRgdzeVLQuMEhd8B8c1zEnkKNkeSTu): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181925296 (635h34m26.053008331s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181941377 (635h34m26.053024412s) [seg_pos={0 0}, upload=556915]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 03:03:26.182003695 (635h34m26.05308673s) [upload=556915]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:03:26.182009416 (635h34m26.053092461s) [upload=556915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:03:26.474026257 (635h34m26.345109292s) [upload=556915]: Done uploading segments... Jan 1 03:03:26.474049391 (635h34m26.345132427s) [upload=556915]: Upload finished. err: <nil> Duration: 57.321356871s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.XX..X.XX.......X...X..X..X...X..X....XXX.X.X..XX.X.XXX..XX.] duration: 5.819506802s trace: https://jaeger.dev.storj.io/trace/579312301588aa45 FAIL (2026-01-01 03:29:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 03:28:37.501792999 (635h59m37.372876034s) [upload=557288]: Starting upload Jan 1 03:28:37.501873622 (635h59m37.372956658s) [upload=557288]: Done starting upload Jan 1 03:28:37.501912718 (635h59m37.372995753s) [upload=557288]: Uploading segments... Jan 1 03:28:37.502049948 (635h59m37.373132983s) [upload=557288]: Got next segment. Inline: false Jan 1 03:28:37.502089003 (635h59m37.373172048s) [upload=557288]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:28:37.502092750 (635h59m37.373175785s) [upload=557288]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:28:37.752742837 (635h59m37.623825872s) [seg_pos={0 0}, upload=557288]: Begin upload segment... Jan 1 03:28:37.752797551 (635h59m37.623880586s) [seg_pos={0 0}, upload=557288]: Done begin upload segment. Jan 1 03:28:37.752848899 (635h59m37.623931934s) [noise=true , address=145.79.227.51:4409 , piece_id=SQ4EOSQSDNDNKSEHDII3THHDARC6KFUP6CIWLR6OSTHMNM6HOBRQ, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.752863076 (635h59m37.623946121s) [noise=true , address=145.79.227.42:4122 , piece_id=JR3I4TEANOK55JT6WUSMZFMYZANF3W4RHVNQZLSFGWB4UJW5EIEA, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.753027038 (635h59m37.624110073s) [noise=true , address=45.91.116.10:4082 , piece_id=CSU66KWPTC2T2GBSIKGVC6FGYN5CFMDMCL5GECFRKHZKUPLE344Q, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.753381403 (635h59m37.624464438s) [noise=true , address=145.79.227.12:4767 , piece_id=TKMTCS2HQIUYNPDBK37P3CO7WZCIRHXWOHWCTCMBXSLPTF3HYT7A, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.753719266 (635h59m37.624802301s) [noise=true , address=145.79.227.11:4731 , piece_id=AB6E35YEK2XXETABVYVZDMAQ7X2MSNPMZVS4HLRCJN3U7C23DDXA, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.754024697 (635h59m37.625107742s) [noise=true , address=145.79.227.6:4259 , piece_id=Q2SUUS4KLKVTQOR5B7JDACBDNC2OZKHM5XH2HP524JJC4UV4CZ2Q, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.754282388 (635h59m37.625365434s) [noise=true , address=45.91.116.6:4974 , piece_id=QH7FF4YUFECLC5Y6K6HJBNNBXFANOEEOZBCCAGAA7V2SE3X6PKLQ, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.754547863 (635h59m37.625630898s) [noise=true , address=77.90.31.19:4842 , piece_id=H6BNUQH7VX5WPSQZFEQJDKWHKCJRATK5EYQTZFVUFLVEYFLQMAFQ, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.754874175 (635h59m37.62595722s) [noise=true , address=45.91.116.5:4297 , piece_id=465KFYYTC3S6RVBFWRUGRUPEMDYKYMFK4V4Y7NLOQIR37P7JAFVA, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.755242065 (635h59m37.6263251s) [noise=true , address=145.79.227.8:4329 , piece_id=ZULYGSQMCGB62PBUBWF5FLXGP6UAQV2ZT4MSXR3VWATIQQWKOCUQ, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.756494821 (635h59m37.627577856s) [noise=true , address=45.91.116.8:4047 , piece_id=4RTHEAZYI4HQSMP2YGM4RSZXAZREO27VVLPY7VRGVHSBCY3PURBQ, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.758165121 (635h59m37.629248156s) [noise=true , address=145.79.227.2:4113 , piece_id=KUVJGHYQ4YR7LAFPOTINTI2PK42ANFPSGHRFRBXVOPQFRW36XZGQ, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.759195824 (635h59m37.630278859s) [noise=true , address=145.79.227.31:4798 , piece_id=M7PPLKAVUD7UJDN5SBRIEQMUPEQ3TCCRZE7OJ7CMRBFFGWDNS4FQ, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.760476342 (635h59m37.631559387s) [noise=true , address=145.79.227.45:4231 , piece_id=CDVCVV3WXHL3NFCAALHSPP5U3KLEUSSVECS65VLGEAEQNMNECY3A, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.762002508 (635h59m37.633085543s) [noise=true , address=77.90.31.10:4517 , piece_id=UMJRZGBF2XLCXBHQLR2WJDUHW5BW25VZIOFFSQMGADHCJGNFY7LQ, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.763280322 (635h59m37.634363357s) [noise=true , address=145.79.227.35:4481 , piece_id=2ZQHNAAWJRJD3IBRIGWYL5TCM5I6ATPDKUWV6ND7ARMFVI4PTVNQ, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:39.457289841 (635h59m39.328372876s) [upload=557288]: Next returned nil segment Jan 1 03:28:39.457302396 (635h59m39.328385441s) [upload=557288]: Waiting for error group managing segments... Jan 1 03:28:39.730758959 (635h59m39.601842004s) [noise=true , address=145.79.227.11:4731 , piece_id=AB6E35YEK2XXETABVYVZDMAQ7X2MSNPMZVS4HLRCJN3U7C23DDXA, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:39.747123368 (635h59m39.618206413s) [noise=true , address=145.79.227.51:4409 , piece_id=SQ4EOSQSDNDNKSEHDII3THHDARC6KFUP6CIWLR6OSTHMNM6HOBRQ, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:39.763070162 (635h59m39.634153258s) [noise=true , address=145.79.227.6:4259 , piece_id=Q2SUUS4KLKVTQOR5B7JDACBDNC2OZKHM5XH2HP524JJC4UV4CZ2Q, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:39.765792416 (635h59m39.636875451s) [noise=true , address=145.79.227.42:4122 , piece_id=JR3I4TEANOK55JT6WUSMZFMYZANF3W4RHVNQZLSFGWB4UJW5EIEA, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:39.769098060 (635h59m39.640181106s) [noise=true , address=145.79.227.8:4329 , piece_id=ZULYGSQMCGB62PBUBWF5FLXGP6UAQV2ZT4MSXR3VWATIQQWKOCUQ, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:39.769254900 (635h59m39.640337935s) [noise=true , address=145.79.227.12:4767 , piece_id=TKMTCS2HQIUYNPDBK37P3CO7WZCIRHXWOHWCTCMBXSLPTF3HYT7A, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:39.772290961 (635h59m39.643373996s) [noise=true , address=145.79.227.2:4113 , piece_id=KUVJGHYQ4YR7LAFPOTINTI2PK42ANFPSGHRFRBXVOPQFRW36XZGQ, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:39.774741276 (635h59m39.645824311s) [noise=true , address=145.79.227.31:4798 , piece_id=M7PPLKAVUD7UJDN5SBRIEQMUPEQ3TCCRZE7OJ7CMRBFFGWDNS4FQ, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:39.780832684 (635h59m39.651915729s) [noise=true , address=145.79.227.45:4231 , piece_id=CDVCVV3WXHL3NFCAALHSPP5U3KLEUSSVECS65VLGEAEQNMNECY3A, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:39.908369770 (635h59m39.779452805s) [noise=true , address=77.90.31.19:4842 , piece_id=H6BNUQH7VX5WPSQZFEQJDKWHKCJRATK5EYQTZFVUFLVEYFLQMAFQ, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:40.081380001 (635h59m39.952463036s) [noise=true , address=45.91.116.6:4974 , piece_id=QH7FF4YUFECLC5Y6K6HJBNNBXFANOEEOZBCCAGAA7V2SE3X6PKLQ, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:40.258939746 (635h59m40.130022791s) [noise=true , address=45.91.116.5:4297 , piece_id=465KFYYTC3S6RVBFWRUGRUPEMDYKYMFK4V4Y7NLOQIR37P7JAFVA, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:40.636022335 (635h59m40.50710537s) [noise=true , address=77.90.31.10:4517 , piece_id=UMJRZGBF2XLCXBHQLR2WJDUHW5BW25VZIOFFSQMGADHCJGNFY7LQ, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:40.636048265 (635h59m40.5071313s) [seg_pos={0 0}, upload=557288]: Segment reached optimal threshold of 13 pieces. Jan 1 03:28:40.636326595 (635h59m40.50740963s) [noise=true , address=45.91.116.10:4082 , piece_id=CSU66KWPTC2T2GBSIKGVC6FGYN5CFMDMCL5GECFRKHZKUPLE344Q, seg_pos={0 0}, upload=557288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:41672->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:41672->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41672->45.91.116.10:4082: use of closed network connection Jan 1 03:28:40.636337966 (635h59m40.507421001s) [noise=true , address=45.91.116.10:4082 , piece_id=CSU66KWPTC2T2GBSIKGVC6FGYN5CFMDMCL5GECFRKHZKUPLE344Q, seg_pos={0 0}, upload=557288]: Long tail cancellation due to optimal threshold Jan 1 03:28:40.636511677 (635h59m40.507594712s) [noise=true , address=45.91.116.8:4047 , piece_id=4RTHEAZYI4HQSMP2YGM4RSZXAZREO27VVLPY7VRGVHSBCY3PURBQ, seg_pos={0 0}, upload=557288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Jan 1 03:28:40.636533639 (635h59m40.507616674s) [noise=true , address=45.91.116.8:4047 , piece_id=4RTHEAZYI4HQSMP2YGM4RSZXAZREO27VVLPY7VRGVHSBCY3PURBQ, seg_pos={0 0}, upload=557288]: Long tail cancellation due to optimal threshold Jan 1 03:28:40.636557795 (635h59m40.50764083s) [noise=true , address=145.79.227.35:4481 , piece_id=2ZQHNAAWJRJD3IBRIGWYL5TCM5I6ATPDKUWV6ND7ARMFVI4PTVNQ, seg_pos={0 0}, upload=557288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Jan 1 03:28:40.636586881 (635h59m40.507669916s) [noise=true , address=145.79.227.35:4481 , piece_id=2ZQHNAAWJRJD3IBRIGWYL5TCM5I6ATPDKUWV6ND7ARMFVI4PTVNQ, seg_pos={0 0}, upload=557288]: Long tail cancellation due to optimal threshold Jan 1 03:28:40.636609183 (635h59m40.507692228s) [seg_pos={0 0}, upload=557288]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:28:40.636721086 (635h59m40.507804131s) [upload=557288]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:28:40.636766863 (635h59m40.507849908s) [upload=557288]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:28:40.929867821 (635h59m40.800950856s) [upload=557288]: Done uploading segments... Jan 1 03:28:40.929886277 (635h59m40.800969312s) [upload=557288]: Upload finished. err: <nil> Duration: 5.819532622s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................X......X..........................] duration: 3.582464478s trace: https://jaeger.dev.storj.io/trace/679257cbe75708e0 FAIL (2026-01-01 03:29:44 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.582484777s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X....X.............................] duration: 50.184723013s trace: https://jaeger.dev.storj.io/trace/725fc16e891a4127 FAIL (2026-01-01 03:29:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== 3:03:23.412681369 (635h34m23.283764404s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413214023 (635h34m23.284297058s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413504245 (635h34m23.28458728s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.413821750 (635h34m23.284904785s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414124316 (635h34m23.285207351s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414417554 (635h34m23.285500589s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.414689172 (635h34m23.285772207s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415023338 (635h34m23.286106373s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415341384 (635h34m23.286424419s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.415659309 (635h34m23.286742354s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416184289 (635h34m23.287267324s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416571025 (635h34m23.28765406s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.416998890 (635h34m23.288081925s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.417376770 (635h34m23.288459815s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.418791664 (635h34m23.289874709s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.420183384 (635h34m23.291266419s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:23.421563200 (635h34m23.292646246s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Uploading piece... Jan 1 03:03:24.823751878 (635h34m24.694834913s) [upload=556915]: Next returned nil segment Jan 1 03:03:24.823765594 (635h34m24.694848629s) [upload=556915]: Waiting for error group managing segments... Jan 1 03:03:25.191603537 (635h34m25.062686582s) [noise=true , address=64.49.13.140:4507 , piece_id=WNAJ4QH56OZWJORRK6ZOPAHFKJXDKDERXZLX6DQD3KIJ46NIE3TA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.271741648 (635h34m25.142824683s) [noise=true , address=64.49.13.137:4497 , piece_id=DFQ324GQ3WOKL6A62S7AGCLKBZB53MLQ6DYV7SRQPQ2O2R7OTUJQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.278781112 (635h34m25.149864157s) [noise=true , address=64.49.13.138:4488 , piece_id=JNH4XJJELUTQIURH3ETGT7VKZ5JC3FIMM7JNHMHDW2275QPKMKQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.337014705 (635h34m25.20809774s) [noise=true , address=64.49.13.155:4917 , piece_id=PE7KE7QXTP4ZGXINBUAQZIEYDBNFMPHCI7AJTCSYT423CTYCHPXQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.372659678 (635h34m25.243742713s) [noise=true , address=145.79.227.49:4365 , piece_id=SO7XSDVKYEX4ECNVADJQDUK64TUT32222TNUBMWFNHC5HZAUM3FA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.439670894 (635h34m25.31075394s) [noise=true , address=145.79.227.29:4968 , piece_id=ZPVITFD7BF7RISLKB2ARKROD3XCN2CQGZAEEQ4KOGG664ECM77MQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.447380494 (635h34m25.318463529s) [noise=true , address=77.90.31.18:4823 , piece_id=IIE2CBN3CH56HHATQPIDWYR6QVKSMTOO3NPWU6JNGJ7YS5WAMLWQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.459093186 (635h34m25.330176231s) [noise=true , address=77.90.31.11:4579 , piece_id=JMPF6GVHX5U2X565WFN6ACFGUJUNIXHIIOYEYAPHN2F2Z57EC4BA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.602592395 (635h34m25.47367543s) [noise=true , address=45.91.116.8:4072 , piece_id=PLQWINOE2IFCKVJC4NJOVZTU57BSMH52KSUDN6C43Q6E7KLE77DA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.719366504 (635h34m25.59044954s) [noise=true , address=45.91.116.12:4230 , piece_id=WQHO3KOUTZSHBZIDUXANBVFSUKTBKEWHTZ3Z7BNGXGOFP2BMJHWA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:25.875237800 (635h34m25.746320845s) [noise=true , address=77.90.31.16:4745 , piece_id=X3346RE3JG67UDVV6RJLDZ4MB4X7YH4MURJLWOSFCXKP6EAKYRZQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.143929177 (635h34m26.015012212s) [noise=true , address=193.149.164.143:4695 , piece_id=VKXG5D4URVMCO7UOKFRFMUYVBBU5UAXSYTXUNV6HHPZLL2YK6KIA, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.159675169 (635h34m26.030758204s) [noise=true , address=193.149.164.143:4701 , piece_id=AX4DX3OLJD4AJCA5ZJCFRZQOQNTSLTUPCHHTZQE5NBEXSDMUE65Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181352455 (635h34m26.05243549s) [noise=true , address=145.79.227.44:4218 , piece_id=7LKHYUAY57DG56OJ725O4YV2FU4LCZ5IIJJSUERKFYPEIKN5UI3Q, seg_pos={0 0}, upload=556915]: Done uploading piece. err: <nil> Jan 1 03:03:26.181379146 (635h34m26.052462181s) [seg_pos={0 0}, upload=556915]: Segment reached optimal threshold of 14 pieces. Jan 1 03:03:26.181716378 (635h34m26.052799413s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Pvm1SftEF3tnCEBqUn1ZmyVcLDBkR6pyegEb7mtzjdQiiqYqt): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181723281 (635h34m26.052806316s) [noise=true , address=145.79.227.30:4777 , piece_id=4KKGADKPPXPM2CDKIENJOE4LQW2YWWFYR4AF64QM7JWHVNSKAQCQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181784638 (635h34m26.052867703s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tvmo3nSgBPcFYW9MmZQGUeMoD3m3ykJHwVMj6tj69iQKgX3Mz): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181848530 (635h34m26.052931565s) [noise=true , address=77.90.31.19:4862 , piece_id=WKTWFMTPH4OG6Y4ITV5RTWQNMTXNJC4F42DGWRUQUZD3CVPUWCYQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181859921 (635h34m26.052942956s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181866474 (635h34m26.052949509s) [noise=true , address=145.79.227.35:4502 , piece_id=YYGKYJUY55TYN7QZDWZTIF24LSPNZUL6VEBO3UF6XM4Q2VF6HMSQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181920486 (635h34m26.053003522s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:184g6qHdezLW3d2CKNW8GRgdzeVLQuMEhd8B8c1zEnkKNkeSTu): protocol: expected piece hash; context canceled; EOF Jan 1 03:03:26.181925296 (635h34m26.053008331s) [noise=true , address=45.91.116.4:4948 , piece_id=TKLEIRFN5JQNMMIU6DTNDY2MHARNFIWQLZBJVJ3FAHGFIRDO6OQQ, seg_pos={0 0}, upload=556915]: Long tail cancellation due to optimal threshold Jan 1 03:03:26.181941377 (635h34m26.053024412s) [seg_pos={0 0}, upload=556915]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jan 1 03:03:26.182003695 (635h34m26.05308673s) [upload=556915]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:03:26.182009416 (635h34m26.053092461s) [upload=556915]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:03:26.474026257 (635h34m26.345109292s) [upload=556915]: Done uploading segments... Jan 1 03:03:26.474049391 (635h34m26.345132427s) [upload=556915]: Upload finished. err: <nil> Duration: 50.184761416s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................X......X.........................] duration: 3.509721908s trace: https://jaeger.dev.storj.io/trace/679257cbe75708e0 FAIL (2026-01-01 03:29:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.509776893s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [..XX..X.XX.......X...X..X..X...X..X....XXX.X.X..XX.X.XXX..XX] duration: 30.005043778s trace: https://jaeger.dev.storj.io/trace/579312301588aa45 FAIL (2026-01-01 03:29:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 1 03:28:37.501792999 (635h59m37.372876034s) [upload=557288]: Starting upload Jan 1 03:28:37.501873622 (635h59m37.372956658s) [upload=557288]: Done starting upload Jan 1 03:28:37.501912718 (635h59m37.372995753s) [upload=557288]: Uploading segments... Jan 1 03:28:37.502049948 (635h59m37.373132983s) [upload=557288]: Got next segment. Inline: false Jan 1 03:28:37.502089003 (635h59m37.373172048s) [upload=557288]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:28:37.502092750 (635h59m37.373175785s) [upload=557288]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:28:37.752742837 (635h59m37.623825872s) [seg_pos={0 0}, upload=557288]: Begin upload segment... Jan 1 03:28:37.752797551 (635h59m37.623880586s) [seg_pos={0 0}, upload=557288]: Done begin upload segment. Jan 1 03:28:37.752848899 (635h59m37.623931934s) [noise=true , address=145.79.227.51:4409 , piece_id=SQ4EOSQSDNDNKSEHDII3THHDARC6KFUP6CIWLR6OSTHMNM6HOBRQ, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.752863076 (635h59m37.623946121s) [noise=true , address=145.79.227.42:4122 , piece_id=JR3I4TEANOK55JT6WUSMZFMYZANF3W4RHVNQZLSFGWB4UJW5EIEA, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.753027038 (635h59m37.624110073s) [noise=true , address=45.91.116.10:4082 , piece_id=CSU66KWPTC2T2GBSIKGVC6FGYN5CFMDMCL5GECFRKHZKUPLE344Q, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.753381403 (635h59m37.624464438s) [noise=true , address=145.79.227.12:4767 , piece_id=TKMTCS2HQIUYNPDBK37P3CO7WZCIRHXWOHWCTCMBXSLPTF3HYT7A, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.753719266 (635h59m37.624802301s) [noise=true , address=145.79.227.11:4731 , piece_id=AB6E35YEK2XXETABVYVZDMAQ7X2MSNPMZVS4HLRCJN3U7C23DDXA, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.754024697 (635h59m37.625107742s) [noise=true , address=145.79.227.6:4259 , piece_id=Q2SUUS4KLKVTQOR5B7JDACBDNC2OZKHM5XH2HP524JJC4UV4CZ2Q, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.754282388 (635h59m37.625365434s) [noise=true , address=45.91.116.6:4974 , piece_id=QH7FF4YUFECLC5Y6K6HJBNNBXFANOEEOZBCCAGAA7V2SE3X6PKLQ, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.754547863 (635h59m37.625630898s) [noise=true , address=77.90.31.19:4842 , piece_id=H6BNUQH7VX5WPSQZFEQJDKWHKCJRATK5EYQTZFVUFLVEYFLQMAFQ, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.754874175 (635h59m37.62595722s) [noise=true , address=45.91.116.5:4297 , piece_id=465KFYYTC3S6RVBFWRUGRUPEMDYKYMFK4V4Y7NLOQIR37P7JAFVA, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.755242065 (635h59m37.6263251s) [noise=true , address=145.79.227.8:4329 , piece_id=ZULYGSQMCGB62PBUBWF5FLXGP6UAQV2ZT4MSXR3VWATIQQWKOCUQ, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.756494821 (635h59m37.627577856s) [noise=true , address=45.91.116.8:4047 , piece_id=4RTHEAZYI4HQSMP2YGM4RSZXAZREO27VVLPY7VRGVHSBCY3PURBQ, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.758165121 (635h59m37.629248156s) [noise=true , address=145.79.227.2:4113 , piece_id=KUVJGHYQ4YR7LAFPOTINTI2PK42ANFPSGHRFRBXVOPQFRW36XZGQ, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.759195824 (635h59m37.630278859s) [noise=true , address=145.79.227.31:4798 , piece_id=M7PPLKAVUD7UJDN5SBRIEQMUPEQ3TCCRZE7OJ7CMRBFFGWDNS4FQ, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.760476342 (635h59m37.631559387s) [noise=true , address=145.79.227.45:4231 , piece_id=CDVCVV3WXHL3NFCAALHSPP5U3KLEUSSVECS65VLGEAEQNMNECY3A, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.762002508 (635h59m37.633085543s) [noise=true , address=77.90.31.10:4517 , piece_id=UMJRZGBF2XLCXBHQLR2WJDUHW5BW25VZIOFFSQMGADHCJGNFY7LQ, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:37.763280322 (635h59m37.634363357s) [noise=true , address=145.79.227.35:4481 , piece_id=2ZQHNAAWJRJD3IBRIGWYL5TCM5I6ATPDKUWV6ND7ARMFVI4PTVNQ, seg_pos={0 0}, upload=557288]: Uploading piece... Jan 1 03:28:39.457289841 (635h59m39.328372876s) [upload=557288]: Next returned nil segment Jan 1 03:28:39.457302396 (635h59m39.328385441s) [upload=557288]: Waiting for error group managing segments... Jan 1 03:28:39.730758959 (635h59m39.601842004s) [noise=true , address=145.79.227.11:4731 , piece_id=AB6E35YEK2XXETABVYVZDMAQ7X2MSNPMZVS4HLRCJN3U7C23DDXA, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:39.747123368 (635h59m39.618206413s) [noise=true , address=145.79.227.51:4409 , piece_id=SQ4EOSQSDNDNKSEHDII3THHDARC6KFUP6CIWLR6OSTHMNM6HOBRQ, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:39.763070162 (635h59m39.634153258s) [noise=true , address=145.79.227.6:4259 , piece_id=Q2SUUS4KLKVTQOR5B7JDACBDNC2OZKHM5XH2HP524JJC4UV4CZ2Q, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:39.765792416 (635h59m39.636875451s) [noise=true , address=145.79.227.42:4122 , piece_id=JR3I4TEANOK55JT6WUSMZFMYZANF3W4RHVNQZLSFGWB4UJW5EIEA, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:39.769098060 (635h59m39.640181106s) [noise=true , address=145.79.227.8:4329 , piece_id=ZULYGSQMCGB62PBUBWF5FLXGP6UAQV2ZT4MSXR3VWATIQQWKOCUQ, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:39.769254900 (635h59m39.640337935s) [noise=true , address=145.79.227.12:4767 , piece_id=TKMTCS2HQIUYNPDBK37P3CO7WZCIRHXWOHWCTCMBXSLPTF3HYT7A, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:39.772290961 (635h59m39.643373996s) [noise=true , address=145.79.227.2:4113 , piece_id=KUVJGHYQ4YR7LAFPOTINTI2PK42ANFPSGHRFRBXVOPQFRW36XZGQ, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:39.774741276 (635h59m39.645824311s) [noise=true , address=145.79.227.31:4798 , piece_id=M7PPLKAVUD7UJDN5SBRIEQMUPEQ3TCCRZE7OJ7CMRBFFGWDNS4FQ, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:39.780832684 (635h59m39.651915729s) [noise=true , address=145.79.227.45:4231 , piece_id=CDVCVV3WXHL3NFCAALHSPP5U3KLEUSSVECS65VLGEAEQNMNECY3A, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:39.908369770 (635h59m39.779452805s) [noise=true , address=77.90.31.19:4842 , piece_id=H6BNUQH7VX5WPSQZFEQJDKWHKCJRATK5EYQTZFVUFLVEYFLQMAFQ, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:40.081380001 (635h59m39.952463036s) [noise=true , address=45.91.116.6:4974 , piece_id=QH7FF4YUFECLC5Y6K6HJBNNBXFANOEEOZBCCAGAA7V2SE3X6PKLQ, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:40.258939746 (635h59m40.130022791s) [noise=true , address=45.91.116.5:4297 , piece_id=465KFYYTC3S6RVBFWRUGRUPEMDYKYMFK4V4Y7NLOQIR37P7JAFVA, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:40.636022335 (635h59m40.50710537s) [noise=true , address=77.90.31.10:4517 , piece_id=UMJRZGBF2XLCXBHQLR2WJDUHW5BW25VZIOFFSQMGADHCJGNFY7LQ, seg_pos={0 0}, upload=557288]: Done uploading piece. err: <nil> Jan 1 03:28:40.636048265 (635h59m40.5071313s) [seg_pos={0 0}, upload=557288]: Segment reached optimal threshold of 13 pieces. Jan 1 03:28:40.636326595 (635h59m40.50740963s) [noise=true , address=45.91.116.10:4082 , piece_id=CSU66KWPTC2T2GBSIKGVC6FGYN5CFMDMCL5GECFRKHZKUPLE344Q, seg_pos={0 0}, upload=557288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:41672->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:41672->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41672->45.91.116.10:4082: use of closed network connection Jan 1 03:28:40.636337966 (635h59m40.507421001s) [noise=true , address=45.91.116.10:4082 , piece_id=CSU66KWPTC2T2GBSIKGVC6FGYN5CFMDMCL5GECFRKHZKUPLE344Q, seg_pos={0 0}, upload=557288]: Long tail cancellation due to optimal threshold Jan 1 03:28:40.636511677 (635h59m40.507594712s) [noise=true , address=45.91.116.8:4047 , piece_id=4RTHEAZYI4HQSMP2YGM4RSZXAZREO27VVLPY7VRGVHSBCY3PURBQ, seg_pos={0 0}, upload=557288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Jan 1 03:28:40.636533639 (635h59m40.507616674s) [noise=true , address=45.91.116.8:4047 , piece_id=4RTHEAZYI4HQSMP2YGM4RSZXAZREO27VVLPY7VRGVHSBCY3PURBQ, seg_pos={0 0}, upload=557288]: Long tail cancellation due to optimal threshold Jan 1 03:28:40.636557795 (635h59m40.50764083s) [noise=true , address=145.79.227.35:4481 , piece_id=2ZQHNAAWJRJD3IBRIGWYL5TCM5I6ATPDKUWV6ND7ARMFVI4PTVNQ, seg_pos={0 0}, upload=557288]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Jan 1 03:28:40.636586881 (635h59m40.507669916s) [noise=true , address=145.79.227.35:4481 , piece_id=2ZQHNAAWJRJD3IBRIGWYL5TCM5I6ATPDKUWV6ND7ARMFVI4PTVNQ, seg_pos={0 0}, upload=557288]: Long tail cancellation due to optimal threshold Jan 1 03:28:40.636609183 (635h59m40.507692228s) [seg_pos={0 0}, upload=557288]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:28:40.636721086 (635h59m40.507804131s) [upload=557288]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:28:40.636766863 (635h59m40.507849908s) [upload=557288]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:28:40.929867821 (635h59m40.800950856s) [upload=557288]: Done uploading segments... Jan 1 03:28:40.929886277 (635h59m40.800969312s) [upload=557288]: Upload finished. err: <nil> Duration: 30.005060228s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X..XX..X.XX.......X...X..X..X...X..X....XXX.X.X..XX.X.XXX..X] duration: 30.00337253s trace: https://jaeger.dev.storj.io/trace/f7c6ac8b2353b49 FAIL (2026-01-01 03:28:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 1 03:28:07.253328587 (635h59m7.124411633s) [upload=557279]: Starting upload Jan 1 03:28:07.253410022 (635h59m7.124493067s) [upload=557279]: Done starting upload Jan 1 03:28:07.253570238 (635h59m7.124653283s) [upload=557279]: Uploading segments... Jan 1 03:28:07.253647274 (635h59m7.124730319s) [upload=557279]: Got next segment. Inline: false Jan 1 03:28:07.253684345 (635h59m7.12476739s) [upload=557279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 1 03:28:07.253690055 (635h59m7.1247731s) [upload=557279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 1 03:28:07.498533777 (635h59m7.369616842s) [seg_pos={0 0}, upload=557279]: Begin upload segment... Jan 1 03:28:07.498593951 (635h59m7.369676986s) [seg_pos={0 0}, upload=557279]: Done begin upload segment. Jan 1 03:28:07.498631613 (635h59m7.369714658s) [noise=true , address=145.79.227.24:4582 , piece_id=6GXSMV7YOXNQ3NZ4EDU7O4WREC6IUMKSXHS2CWN2I2XQDMG45BHQ, seg_pos={0 0}, upload=557279]: Uploading piece... Jan 1 03:28:07.498641241 (635h59m7.369724286s) [noise=true , address=145.79.227.3:4151 , piece_id=PUPPARLZZYOEYPPH3NDVWCVOSMPPSARYMT2HD7FON5CG775O5XLA, seg_pos={0 0}, upload=557279]: Uploading piece... Jan 1 03:28:07.499743530 (635h59m7.370826575s) [noise=true , address=145.79.227.7:4294 , piece_id=7ULPLVZHEDBODLGNDBPWNVV6GO4EMPR3SBRYV3IQMROOBFFQN67Q, seg_pos={0 0}, upload=557279]: Uploading piece... Jan 1 03:28:07.499983757 (635h59m7.371066802s) [noise=true , address=145.79.227.35:4481 , piece_id=NG7U2HU5E4I7YXCXT6OQRFRJNVSSXA72IA44I54RBRRFIZ5BFEBQ, seg_pos={0 0}, upload=557279]: Uploading piece... Jan 1 03:28:07.500540567 (635h59m7.371623602s) [noise=true , address=145.79.227.11:4729 , piece_id=RWDMZ3YI73NCFRESLRJCALHCB6BORRZNG6XIGLI2LL4CGUK56P2A, seg_pos={0 0}, upload=557279]: Uploading piece... Jan 1 03:28:07.500772429 (635h59m7.371855474s) [noise=true , address=45.91.116.9:4261 , piece_id=B6LF3ISFK7TXIFS4ZRVC4E773DUAIHAP4U7CION5YZWHVGNDDLUA, seg_pos={0 0}, upload=557279]: Uploading piece... Jan 1 03:28:07.501723570 (635h59m7.372806615s) [noise=true , address=45.91.116.11:4117 , piece_id=LCIQQZMYUVRQYQ545KW5F2KR23DEABPKYSBVA2PBIYAFEFF36BJA, seg_pos={0 0}, upload=557279]: Uploading piece... Jan 1 03:28:07.502829776 (635h59m7.373912832s) [noise=true , address=45.91.116.6:4975 , piece_id=IJ3QMOH67EXJ5633DF66XS3O6EUMTPIQ7GWWYYWG4NSFXN5NRGZA, seg_pos={0 0}, upload=557279]: Uploading piece... Jan 1 03:28:07.503605684 (635h59m7.374688729s) [noise=true , address=45.91.116.13:4153 , piece_id=PZXELTC4QPLDQ4GPA2MNHOMTXO7PA7VTE4W4TNTVWORK7FIK62VQ, seg_pos={0 0}, upload=557279]: Uploading piece... Jan 1 03:28:07.504655633 (635h59m7.375738678s) [noise=true , address=145.79.227.10:4402 , piece_id=3HK7TUZZUV3FA6L34WL5VPUE7G7ZCTWR7UOLY5H5JIAN26HIJLWQ, seg_pos={0 0}, upload=557279]: Uploading piece... Jan 1 03:28:07.505412875 (635h59m7.37649591s) [noise=true , address=145.79.227.34:4905 , piece_id=5XTPULV4RIMNGGE26RGD5LY65LJ66UOHFDWMAKJLOFDVM5VZ72EA, seg_pos={0 0}, upload=557279]: Uploading piece... Jan 1 03:28:07.506990188 (635h59m7.378073223s) [noise=true , address=145.79.227.37:4566 , piece_id=LX7NGLVZJJA5GXU65NZ7Q4TZOPVT6ILQYW4KRYHZASODUQE2J34Q, seg_pos={0 0}, upload=557279]: Uploading piece... Jan 1 03:28:07.507498326 (635h59m7.378581371s) [noise=true , address=77.90.31.10:4518 , piece_id=ENGLHTHIDGNZ6VEAZIE4TCG77RGUPON6YHCH2VG6BAPGAW3F42BA, seg_pos={0 0}, upload=557279]: Uploading piece... Jan 1 03:28:07.508004549 (635h59m7.379087585s) [noise=true , address=145.79.227.51:4411 , piece_id=IMIY2DCLIF5ECUKEZ6VJB2UH2YP336FKJ7RJN5B6XROJJ7JEZATQ, seg_pos={0 0}, upload=557279]: Uploading piece... Jan 1 03:28:07.508173061 (635h59m7.379256096s) [noise=true , address=145.79.227.15:4874 , piece_id=NV4BZLG5AO7PR6ZJCVSW2OWZ4LQX5MKOWRRYKFXREO66T2JM3RJQ, seg_pos={0 0}, upload=557279]: Uploading piece... Jan 1 03:28:07.508658495 (635h59m7.37974154s) [noise=true , address=45.91.116.10:4082 , piece_id=ESYUWV2JMPEC5CMZYSMFUNTVGPXXWZU2JDYE4OXJMLFLQDNJIO6Q, seg_pos={0 0}, upload=557279]: Uploading piece... Jan 1 03:28:09.118285663 (635h59m8.989368708s) [upload=557279]: Next returned nil segment Jan 1 03:28:09.118300241 (635h59m8.989383286s) [upload=557279]: Waiting for error group managing segments... Jan 1 03:28:09.495428336 (635h59m9.366511371s) [noise=true , address=145.79.227.51:4411 , piece_id=IMIY2DCLIF5ECUKEZ6VJB2UH2YP336FKJ7RJN5B6XROJJ7JEZATQ, seg_pos={0 0}, upload=557279]: Done uploading piece. err: <nil> Jan 1 03:28:09.497043622 (635h59m9.368126657s) [noise=true , address=145.79.227.24:4582 , piece_id=6GXSMV7YOXNQ3NZ4EDU7O4WREC6IUMKSXHS2CWN2I2XQDMG45BHQ, seg_pos={0 0}, upload=557279]: Done uploading piece. err: <nil> Jan 1 03:28:09.502819200 (635h59m9.373902245s) [noise=true , address=145.79.227.15:4874 , piece_id=NV4BZLG5AO7PR6ZJCVSW2OWZ4LQX5MKOWRRYKFXREO66T2JM3RJQ, seg_pos={0 0}, upload=557279]: Done uploading piece. err: <nil> Jan 1 03:28:09.509144974 (635h59m9.380228019s) [noise=true , address=145.79.227.3:4151 , piece_id=PUPPARLZZYOEYPPH3NDVWCVOSMPPSARYMT2HD7FON5CG775O5XLA, seg_pos={0 0}, upload=557279]: Done uploading piece. err: <nil> Jan 1 03:28:09.520689737 (635h59m9.391772772s) [noise=true , address=145.79.227.37:4566 , piece_id=LX7NGLVZJJA5GXU65NZ7Q4TZOPVT6ILQYW4KRYHZASODUQE2J34Q, seg_pos={0 0}, upload=557279]: Done uploading piece. err: <nil> Jan 1 03:28:09.564230844 (635h59m9.435313879s) [noise=true , address=145.79.227.10:4402 , piece_id=3HK7TUZZUV3FA6L34WL5VPUE7G7ZCTWR7UOLY5H5JIAN26HIJLWQ, seg_pos={0 0}, upload=557279]: Done uploading piece. err: <nil> Jan 1 03:28:09.576782494 (635h59m9.447865529s) [noise=true , address=145.79.227.7:4294 , piece_id=7ULPLVZHEDBODLGNDBPWNVV6GO4EMPR3SBRYV3IQMROOBFFQN67Q, seg_pos={0 0}, upload=557279]: Done uploading piece. err: <nil> Jan 1 03:28:09.581008400 (635h59m9.452091435s) [noise=true , address=145.79.227.11:4729 , piece_id=RWDMZ3YI73NCFRESLRJCALHCB6BORRZNG6XIGLI2LL4CGUK56P2A, seg_pos={0 0}, upload=557279]: Done uploading piece. err: <nil> Jan 1 03:28:09.710474799 (635h59m9.581557834s) [noise=true , address=145.79.227.35:4481 , piece_id=NG7U2HU5E4I7YXCXT6OQRFRJNVSSXA72IA44I54RBRRFIZ5BFEBQ, seg_pos={0 0}, upload=557279]: Done uploading piece. err: <nil> Jan 1 03:28:09.803721181 (635h59m9.674804216s) [noise=true , address=45.91.116.9:4261 , piece_id=B6LF3ISFK7TXIFS4ZRVC4E773DUAIHAP4U7CION5YZWHVGNDDLUA, seg_pos={0 0}, upload=557279]: Done uploading piece. err: <nil> Jan 1 03:28:09.806404951 (635h59m9.677487996s) [noise=true , address=45.91.116.10:4082 , piece_id=ESYUWV2JMPEC5CMZYSMFUNTVGPXXWZU2JDYE4OXJMLFLQDNJIO6Q, seg_pos={0 0}, upload=557279]: Done uploading piece. err: <nil> Jan 1 03:28:09.826098480 (635h59m9.697181515s) [noise=true , address=45.91.116.13:4153 , piece_id=PZXELTC4QPLDQ4GPA2MNHOMTXO7PA7VTE4W4TNTVWORK7FIK62VQ, seg_pos={0 0}, upload=557279]: Done uploading piece. err: <nil> Jan 1 03:28:09.910685474 (635h59m9.781768509s) [noise=true , address=45.91.116.6:4975 , piece_id=IJ3QMOH67EXJ5633DF66XS3O6EUMTPIQ7GWWYYWG4NSFXN5NRGZA, seg_pos={0 0}, upload=557279]: Done uploading piece. err: <nil> Jan 1 03:28:09.910705472 (635h59m9.781788507s) [seg_pos={0 0}, upload=557279]: Segment reached optimal threshold of 13 pieces. Jan 1 03:28:09.910950689 (635h59m9.782033734s) [noise=true , address=77.90.31.10:4518 , piece_id=ENGLHTHIDGNZ6VEAZIE4TCG77RGUPON6YHCH2VG6BAPGAW3F42BA, seg_pos={0 0}, upload=557279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Jan 1 03:28:09.910959706 (635h59m9.782042741s) [noise=true , address=77.90.31.10:4518 , piece_id=ENGLHTHIDGNZ6VEAZIE4TCG77RGUPON6YHCH2VG6BAPGAW3F42BA, seg_pos={0 0}, upload=557279]: Long tail cancellation due to optimal threshold Jan 1 03:28:09.911151171 (635h59m9.782234216s) [noise=true , address=145.79.227.34:4905 , piece_id=5XTPULV4RIMNGGE26RGD5LY65LJ66UOHFDWMAKJLOFDVM5VZ72EA, seg_pos={0 0}, upload=557279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Jan 1 03:28:09.911158846 (635h59m9.782241891s) [noise=true , address=145.79.227.34:4905 , piece_id=5XTPULV4RIMNGGE26RGD5LY65LJ66UOHFDWMAKJLOFDVM5VZ72EA, seg_pos={0 0}, upload=557279]: Long tail cancellation due to optimal threshold Jan 1 03:28:09.911333638 (635h59m9.782416713s) [noise=true , address=45.91.116.11:4117 , piece_id=LCIQQZMYUVRQYQ545KW5F2KR23DEABPKYSBVA2PBIYAFEFF36BJA, seg_pos={0 0}, upload=557279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: read tcp 188.166.225.170:48982->45.91.116.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 1 03:28:09.911353586 (635h59m9.782436631s) [noise=true , address=45.91.116.11:4117 , piece_id=LCIQQZMYUVRQYQ545KW5F2KR23DEABPKYSBVA2PBIYAFEFF36BJA, seg_pos={0 0}, upload=557279]: Long tail cancellation due to optimal threshold Jan 1 03:28:09.911389895 (635h59m9.78247294s) [seg_pos={0 0}, upload=557279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 1 03:28:09.911527868 (635h59m9.782610913s) [upload=557279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 1 03:28:09.911535864 (635h59m9.782618909s) [upload=557279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 1 03:28:10.198152754 (635h59m10.069235819s) [upload=557279]: Done uploading segments... Jan 1 03:28:10.198176138 (635h59m10.069259183s) [upload=557279]: Upload finished. err: <nil> Duration: 30.003395915s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)