Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X................XX........................................] duration: 48.015429014s trace: https://jaeger.dev.storj.io/trace/9fd07cafe8f934a FAIL (2025-12-27 16:22:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315411726 (528h16m20.186494771s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315905206 (528h16m20.186988251s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316356304 (528h16m20.187439339s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316793537 (528h16m20.187876582s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.317233315 (528h16m20.18831636s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318042255 (528h16m20.1891253s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318150922 (528h16m20.189233967s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318513712 (528h16m20.189596747s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318955464 (528h16m20.190038509s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319037570 (528h16m20.190120605s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319389541 (528h16m20.190472576s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319874033 (528h16m20.190957078s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320378844 (528h16m20.191461879s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320898604 (528h16m20.191981649s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.321434254 (528h16m20.192517289s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:21.620856000 (528h16m21.491939035s) [upload=463344]: Next returned nil segment Dec 27 15:45:21.620875698 (528h16m21.491958743s) [upload=463344]: Waiting for error group managing segments... Dec 27 15:45:21.953178480 (528h16m21.824261516s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.194632602 (528h16m22.065715647s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.195118968 (528h16m22.066202013s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.204573761 (528h16m22.075656806s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.213413343 (528h16m22.084496388s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.372654386 (528h16m22.243737431s) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.473126101 (528h16m22.344209146s) [noise=true , address=77.90.31.12:4615 , piece_id=NTYP2JTUJYSBE4NBMQHTEAY2DA333CR7V3622PVQ3OGRMSPCJK3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.484926481 (528h16m22.356009536s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.563981276 (528h16m22.435064311s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.893470090 (528h16m22.764553126s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.105921755 (528h16m22.9770048s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.153398930 (528h16m23.024481975s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.218328398 (528h16m23.089411443s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244693782 (528h16m23.115776827s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244714231 (528h16m23.115797266s) [seg_pos={0 0}, upload=463344]: Segment reached optimal threshold of 14 pieces. Dec 27 15:45:23.244907037 (528h16m23.115990073s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.244913159 (528h16m23.115996204s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245080929 (528h16m23.116163964s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245087220 (528h16m23.116170275s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245280880 (528h16m23.116363925s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection Dec 27 15:45:23.245370850 (528h16m23.116453885s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245453588 (528h16m23.116536644s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f2MspBPykwYCWAYzJ9PCAXop6mPNarJt3fa6fWZoLipZZQdy1): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245474488 (528h16m23.116557533s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245514535 (528h16m23.11659757s) [seg_pos={0 0}, upload=463344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:45:23.246027922 (528h16m23.117110977s) [upload=463344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:45:23.246044133 (528h16m23.117127168s) [upload=463344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:45:23.560988923 (528h16m23.432071968s) [upload=463344]: Done uploading segments... Dec 27 15:45:23.561009573 (528h16m23.432092618s) [upload=463344]: Upload finished. err: <nil> Duration: 48.01544774s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [XXX....XXXX..XX..XX......X......XX...X.............X.....X..] duration: 4.78187073s trace: https://jaeger.dev.storj.io/trace/5484ae4968b2b854 FAIL (2025-12-27 16:22:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.781898564s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [XXXX....XXXX..XX..XX......X......XX...X.............X.....X.] duration: 3.842097297s trace: https://jaeger.dev.storj.io/trace/5484ae4968b2b854 FAIL (2025-12-27 16:22:12 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.842121733s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X...........X.......X..X.XX..X..X....XX.X....X..X.XXX..X.XX.] duration: 5.79799011s trace: https://jaeger.dev.storj.io/trace/71c0dd91385fa63a FAIL (2025-12-27 16:22:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: TYWZQDLCTAH3UISSSRGDYZKYKSQPZWARWW6ABTWR7VPSGOJJEQYA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: PIFBU266FRJVQNLAK6I5B3PEX2MQMHJRLYYYNEJQH2XU5FEQMW2A) context deadline exceeded ======LOGS====== Dec 27 16:21:16.302137089 (528h52m16.173220124s) [upload=463864]: Starting upload Dec 27 16:21:16.302252789 (528h52m16.173335834s) [upload=463864]: Done starting upload Dec 27 16:21:16.302312974 (528h52m16.173396019s) [upload=463864]: Uploading segments... Dec 27 16:21:16.302486825 (528h52m16.17356987s) [upload=463864]: Got next segment. Inline: false Dec 27 16:21:16.302550677 (528h52m16.173633732s) [upload=463864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:21:16.302557660 (528h52m16.173640705s) [upload=463864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:21:16.549996326 (528h52m16.421079372s) [seg_pos={0 0}, upload=463864]: Begin upload segment... Dec 27 16:21:16.550057193 (528h52m16.421140238s) [seg_pos={0 0}, upload=463864]: Done begin upload segment. Dec 27 16:21:16.550075538 (528h52m16.421158583s) [noise=true , address=145.79.227.39:4015 , piece_id=TFLQCX6H3TM4SHWL4SFR63PSAPFC4YTK47PIKL7A765TO63AOAKA, seg_pos={0 0}, upload=463864]: Uploading piece... Dec 27 16:21:16.550150741 (528h52m16.421233786s) [noise=true , address=77.90.31.21:4877 , piece_id=DFHBRNGPREJMZKZZ7WYZT6W6D4EWWYG7W2F4WK45U6GTYPADEYTA, seg_pos={0 0}, upload=463864]: Uploading piece... Dec 27 16:21:16.550286500 (528h52m16.421369545s) [noise=true , address=145.79.227.36:4518 , piece_id=EN7EVBMHHKZ3DMBBZW3NPKY64ZDFYFCERRCGOYRUJOW6LWGZVNVQ, seg_pos={0 0}, upload=463864]: Uploading piece... Dec 27 16:21:16.550846656 (528h52m16.421929701s) [noise=true , address=145.79.227.16:4912 , piece_id=TYUPK4HDXYUY5KFRBLVBI3VGEOBCFOKSS52TH3QCXGUKZVCYTRYQ, seg_pos={0 0}, upload=463864]: Uploading piece... Dec 27 16:21:16.551306401 (528h52m16.422389447s) [noise=true , address=145.79.227.51:4409 , piece_id=A27IAXSG2ECL4SS3Q4JKUOUXV4IXXKYFMMVVQMM4ZCACYQHMPVQQ, seg_pos={0 0}, upload=463864]: Uploading piece... Dec 27 16:21:16.551693078 (528h52m16.422776123s) [noise=true , address=145.79.227.38:4978 , piece_id=GA3LEYBS4JWYRAPIEF7TK4TIUQFJ5CITT7XLTK5FVRIK36JUIZ4Q, seg_pos={0 0}, upload=463864]: Uploading piece... Dec 27 16:21:16.552103389 (528h52m16.423186424s) [noise=true , address=145.79.227.42:4123 , piece_id=NCAJT4Z3MEDJ6YHV6KSKDGEKMI5W2KFTQP46YTBK2NFHZ2QIXYMA, seg_pos={0 0}, upload=463864]: Uploading piece... Dec 27 16:21:16.552505736 (528h52m16.423588771s) [noise=true , address=145.79.227.47:4267 , piece_id=SFQRNTSMRLPK3YGDZ6EP5DK5A66YUCJQSSYV6K7Z6VSKJ2BON52A, seg_pos={0 0}, upload=463864]: Uploading piece... Dec 27 16:21:16.552963577 (528h52m16.424046622s) [noise=true , address=45.91.116.10:4083 , piece_id=PIFBU266FRJVQNLAK6I5B3PEX2MQMHJRLYYYNEJQH2XU5FEQMW2A, seg_pos={0 0}, upload=463864]: Uploading piece... Dec 27 16:21:16.555154569 (528h52m16.426237615s) [noise=true , address=45.91.116.4:4937 , piece_id=MGGCASLJGRK3X7Q5EAATGZEGQXYEA5HTETC245RC2QNWSVHJDJ5Q, seg_pos={0 0}, upload=463864]: Uploading piece... Dec 27 16:21:16.555731958 (528h52m16.426814993s) [noise=true , address=145.79.227.43:4158 , piece_id=MY4VJBFMA3QL5P5JXSN3YY5HKNEK2FFNASJLDWWRIMOVL5POCKUA, seg_pos={0 0}, upload=463864]: Uploading piece... Dec 27 16:21:16.557946536 (528h52m16.429029581s) [noise=true , address=145.79.227.23:4545 , piece_id=4GDHTKFHLWDZI6NOE2HY3DZFT6GHQSF3OJIDM3CLB57VC37L4B5Q, seg_pos={0 0}, upload=463864]: Uploading piece... Dec 27 16:21:16.559917167 (528h52m16.431000213s) [noise=true , address=145.79.227.12:4767 , piece_id=BPVIVTGUU4SI53J4Z5EAYOCM3Z2E6YLDTLXYATTCRYOMENOS34DQ, seg_pos={0 0}, upload=463864]: Uploading piece... Dec 27 16:21:16.562848801 (528h52m16.433931846s) [noise=true , address=77.90.31.10:4518 , piece_id=YSPOY4DBCBURRYP2B4ZO4OFJYMEQYY6WXT4CKWIDWAFZTTHM4TTQ, seg_pos={0 0}, upload=463864]: Uploading piece... Dec 27 16:21:16.564827699 (528h52m16.435910754s) [noise=true , address=145.79.227.7:4295 , piece_id=UD46ASUVPMUGHRU4U66EQW5TP4WRGW2UNXQMMWRFWUOEC3CV7VBQ, seg_pos={0 0}, upload=463864]: Uploading piece... Dec 27 16:21:16.565203404 (528h52m16.436286449s) [noise=true , address=45.91.116.11:4118 , piece_id=TYWZQDLCTAH3UISSSRGDYZKYKSQPZWARWW6ABTWR7VPSGOJJEQYA, seg_pos={0 0}, upload=463864]: Uploading piece... Dec 27 16:21:18.274659468 (528h52m18.145742513s) [upload=463864]: Next returned nil segment Dec 27 16:21:18.274671781 (528h52m18.145754826s) [upload=463864]: Waiting for error group managing segments... Dec 27 16:21:18.550836227 (528h52m18.421919272s) [noise=true , address=145.79.227.38:4978 , piece_id=GA3LEYBS4JWYRAPIEF7TK4TIUQFJ5CITT7XLTK5FVRIK36JUIZ4Q, seg_pos={0 0}, upload=463864]: Done uploading piece. err: <nil> Dec 27 16:21:18.559524060 (528h52m18.430607095s) [noise=true , address=145.79.227.36:4518 , piece_id=EN7EVBMHHKZ3DMBBZW3NPKY64ZDFYFCERRCGOYRUJOW6LWGZVNVQ, seg_pos={0 0}, upload=463864]: Done uploading piece. err: <nil> Dec 27 16:21:18.562506048 (528h52m18.433589094s) [noise=true , address=145.79.227.42:4123 , piece_id=NCAJT4Z3MEDJ6YHV6KSKDGEKMI5W2KFTQP46YTBK2NFHZ2QIXYMA, seg_pos={0 0}, upload=463864]: Done uploading piece. err: <nil> Dec 27 16:21:18.566186567 (528h52m18.437269612s) [noise=true , address=145.79.227.16:4912 , piece_id=TYUPK4HDXYUY5KFRBLVBI3VGEOBCFOKSS52TH3QCXGUKZVCYTRYQ, seg_pos={0 0}, upload=463864]: Done uploading piece. err: <nil> Dec 27 16:21:18.569756865 (528h52m18.4408399s) [noise=true , address=145.79.227.12:4767 , piece_id=BPVIVTGUU4SI53J4Z5EAYOCM3Z2E6YLDTLXYATTCRYOMENOS34DQ, seg_pos={0 0}, upload=463864]: Done uploading piece. err: <nil> Dec 27 16:21:18.570466457 (528h52m18.441549492s) [noise=true , address=145.79.227.39:4015 , piece_id=TFLQCX6H3TM4SHWL4SFR63PSAPFC4YTK47PIKL7A765TO63AOAKA, seg_pos={0 0}, upload=463864]: Done uploading piece. err: <nil> Dec 27 16:21:18.570896956 (528h52m18.441980001s) [noise=true , address=145.79.227.51:4409 , piece_id=A27IAXSG2ECL4SS3Q4JKUOUXV4IXXKYFMMVVQMM4ZCACYQHMPVQQ, seg_pos={0 0}, upload=463864]: Done uploading piece. err: <nil> Dec 27 16:21:18.571305725 (528h52m18.44238876s) [noise=true , address=145.79.227.47:4267 , piece_id=SFQRNTSMRLPK3YGDZ6EP5DK5A66YUCJQSSYV6K7Z6VSKJ2BON52A, seg_pos={0 0}, upload=463864]: Done uploading piece. err: <nil> Dec 27 16:21:18.573260006 (528h52m18.444343051s) [noise=true , address=145.79.227.7:4295 , piece_id=UD46ASUVPMUGHRU4U66EQW5TP4WRGW2UNXQMMWRFWUOEC3CV7VBQ, seg_pos={0 0}, upload=463864]: Done uploading piece. err: <nil> Dec 27 16:21:18.574780081 (528h52m18.445863126s) [noise=true , address=145.79.227.23:4545 , piece_id=4GDHTKFHLWDZI6NOE2HY3DZFT6GHQSF3OJIDM3CLB57VC37L4B5Q, seg_pos={0 0}, upload=463864]: Done uploading piece. err: <nil> Dec 27 16:21:18.860524047 (528h52m18.731607092s) [noise=true , address=77.90.31.10:4518 , piece_id=YSPOY4DBCBURRYP2B4ZO4OFJYMEQYY6WXT4CKWIDWAFZTTHM4TTQ, seg_pos={0 0}, upload=463864]: Done uploading piece. err: <nil> Dec 27 16:21:18.923855732 (528h52m18.794938767s) [noise=true , address=45.91.116.11:4118 , piece_id=TYWZQDLCTAH3UISSSRGDYZKYKSQPZWARWW6ABTWR7VPSGOJJEQYA, seg_pos={0 0}, upload=463864]: Done uploading piece. err: <nil> Dec 27 16:21:18.984066784 (528h52m18.855149829s) [noise=true , address=45.91.116.10:4083 , piece_id=PIFBU266FRJVQNLAK6I5B3PEX2MQMHJRLYYYNEJQH2XU5FEQMW2A, seg_pos={0 0}, upload=463864]: Done uploading piece. err: <nil> Dec 27 16:21:18.984091040 (528h52m18.855174095s) [seg_pos={0 0}, upload=463864]: Segment reached optimal threshold of 13 pieces. Dec 27 16:21:18.984331538 (528h52m18.855414583s) [noise=true , address=45.91.116.4:4937 , piece_id=MGGCASLJGRK3X7Q5EAATGZEGQXYEA5HTETC245RC2QNWSVHJDJ5Q, seg_pos={0 0}, upload=463864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 27 16:21:18.984340265 (528h52m18.85542331s) [noise=true , address=45.91.116.4:4937 , piece_id=MGGCASLJGRK3X7Q5EAATGZEGQXYEA5HTETC245RC2QNWSVHJDJ5Q, seg_pos={0 0}, upload=463864]: Long tail cancellation due to optimal threshold Dec 27 16:21:18.984450224 (528h52m18.855533259s) [noise=true , address=145.79.227.43:4158 , piece_id=MY4VJBFMA3QL5P5JXSN3YY5HKNEK2FFNASJLDWWRIMOVL5POCKUA, seg_pos={0 0}, upload=463864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 27 16:21:18.984474521 (528h52m18.855557556s) [noise=true , address=145.79.227.43:4158 , piece_id=MY4VJBFMA3QL5P5JXSN3YY5HKNEK2FFNASJLDWWRIMOVL5POCKUA, seg_pos={0 0}, upload=463864]: Long tail cancellation due to optimal threshold Dec 27 16:21:18.984705952 (528h52m18.855788987s) [noise=true , address=77.90.31.21:4877 , piece_id=DFHBRNGPREJMZKZZ7WYZT6W6D4EWWYG7W2F4WK45U6GTYPADEYTA, seg_pos={0 0}, upload=463864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: context canceled; context canceled Dec 27 16:21:18.984716040 (528h52m18.855799086s) [noise=true , address=77.90.31.21:4877 , piece_id=DFHBRNGPREJMZKZZ7WYZT6W6D4EWWYG7W2F4WK45U6GTYPADEYTA, seg_pos={0 0}, upload=463864]: Long tail cancellation due to optimal threshold Dec 27 16:21:18.984763180 (528h52m18.855846225s) [seg_pos={0 0}, upload=463864]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:21:18.984872568 (528h52m18.855955603s) [upload=463864]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:21:18.984879582 (528h52m18.855962617s) [upload=463864]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:21:19.279731311 (528h52m19.150814346s) [upload=463864]: Done uploading segments... Dec 27 16:21:19.279754806 (528h52m19.150837851s) [upload=463864]: Upload finished. err: <nil> Duration: 5.798049633s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XX...........X.......X..X.XX..X..X....XX.X....X..X.XXX..X.XX] duration: 30.002451965s trace: https://jaeger.dev.storj.io/trace/71c0dd91385fa63a FAIL (2025-12-27 16:21:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: TYWZQDLCTAH3UISSSRGDYZKYKSQPZWARWW6ABTWR7VPSGOJJEQYA) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: PIFBU266FRJVQNLAK6I5B3PEX2MQMHJRLYYYNEJQH2XU5FEQMW2A) context deadline exceeded ======LOGS====== Dec 27 16:21:16.302137089 (528h52m16.173220124s) [upload=463864]: Starting upload Dec 27 16:21:16.302252789 (528h52m16.173335834s) [upload=463864]: Done starting upload Dec 27 16:21:16.302312974 (528h52m16.173396019s) [upload=463864]: Uploading segments... Dec 27 16:21:16.302486825 (528h52m16.17356987s) [upload=463864]: Got next segment. Inline: false Dec 27 16:21:16.302550677 (528h52m16.173633732s) [upload=463864]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:21:16.302557660 (528h52m16.173640705s) [upload=463864]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:21:16.549996326 (528h52m16.421079372s) [seg_pos={0 0}, upload=463864]: Begin upload segment... Dec 27 16:21:16.550057193 (528h52m16.421140238s) [seg_pos={0 0}, upload=463864]: Done begin upload segment. Dec 27 16:21:16.550075538 (528h52m16.421158583s) [noise=true , address=145.79.227.39:4015 , piece_id=TFLQCX6H3TM4SHWL4SFR63PSAPFC4YTK47PIKL7A765TO63AOAKA, seg_pos={0 0}, upload=463864]: Uploading piece... Dec 27 16:21:16.550150741 (528h52m16.421233786s) [noise=true , address=77.90.31.21:4877 , piece_id=DFHBRNGPREJMZKZZ7WYZT6W6D4EWWYG7W2F4WK45U6GTYPADEYTA, seg_pos={0 0}, upload=463864]: Uploading piece... Dec 27 16:21:16.550286500 (528h52m16.421369545s) [noise=true , address=145.79.227.36:4518 , piece_id=EN7EVBMHHKZ3DMBBZW3NPKY64ZDFYFCERRCGOYRUJOW6LWGZVNVQ, seg_pos={0 0}, upload=463864]: Uploading piece... Dec 27 16:21:16.550846656 (528h52m16.421929701s) [noise=true , address=145.79.227.16:4912 , piece_id=TYUPK4HDXYUY5KFRBLVBI3VGEOBCFOKSS52TH3QCXGUKZVCYTRYQ, seg_pos={0 0}, upload=463864]: Uploading piece... Dec 27 16:21:16.551306401 (528h52m16.422389447s) [noise=true , address=145.79.227.51:4409 , piece_id=A27IAXSG2ECL4SS3Q4JKUOUXV4IXXKYFMMVVQMM4ZCACYQHMPVQQ, seg_pos={0 0}, upload=463864]: Uploading piece... Dec 27 16:21:16.551693078 (528h52m16.422776123s) [noise=true , address=145.79.227.38:4978 , piece_id=GA3LEYBS4JWYRAPIEF7TK4TIUQFJ5CITT7XLTK5FVRIK36JUIZ4Q, seg_pos={0 0}, upload=463864]: Uploading piece... Dec 27 16:21:16.552103389 (528h52m16.423186424s) [noise=true , address=145.79.227.42:4123 , piece_id=NCAJT4Z3MEDJ6YHV6KSKDGEKMI5W2KFTQP46YTBK2NFHZ2QIXYMA, seg_pos={0 0}, upload=463864]: Uploading piece... Dec 27 16:21:16.552505736 (528h52m16.423588771s) [noise=true , address=145.79.227.47:4267 , piece_id=SFQRNTSMRLPK3YGDZ6EP5DK5A66YUCJQSSYV6K7Z6VSKJ2BON52A, seg_pos={0 0}, upload=463864]: Uploading piece... Dec 27 16:21:16.552963577 (528h52m16.424046622s) [noise=true , address=45.91.116.10:4083 , piece_id=PIFBU266FRJVQNLAK6I5B3PEX2MQMHJRLYYYNEJQH2XU5FEQMW2A, seg_pos={0 0}, upload=463864]: Uploading piece... Dec 27 16:21:16.555154569 (528h52m16.426237615s) [noise=true , address=45.91.116.4:4937 , piece_id=MGGCASLJGRK3X7Q5EAATGZEGQXYEA5HTETC245RC2QNWSVHJDJ5Q, seg_pos={0 0}, upload=463864]: Uploading piece... Dec 27 16:21:16.555731958 (528h52m16.426814993s) [noise=true , address=145.79.227.43:4158 , piece_id=MY4VJBFMA3QL5P5JXSN3YY5HKNEK2FFNASJLDWWRIMOVL5POCKUA, seg_pos={0 0}, upload=463864]: Uploading piece... Dec 27 16:21:16.557946536 (528h52m16.429029581s) [noise=true , address=145.79.227.23:4545 , piece_id=4GDHTKFHLWDZI6NOE2HY3DZFT6GHQSF3OJIDM3CLB57VC37L4B5Q, seg_pos={0 0}, upload=463864]: Uploading piece... Dec 27 16:21:16.559917167 (528h52m16.431000213s) [noise=true , address=145.79.227.12:4767 , piece_id=BPVIVTGUU4SI53J4Z5EAYOCM3Z2E6YLDTLXYATTCRYOMENOS34DQ, seg_pos={0 0}, upload=463864]: Uploading piece... Dec 27 16:21:16.562848801 (528h52m16.433931846s) [noise=true , address=77.90.31.10:4518 , piece_id=YSPOY4DBCBURRYP2B4ZO4OFJYMEQYY6WXT4CKWIDWAFZTTHM4TTQ, seg_pos={0 0}, upload=463864]: Uploading piece... Dec 27 16:21:16.564827699 (528h52m16.435910754s) [noise=true , address=145.79.227.7:4295 , piece_id=UD46ASUVPMUGHRU4U66EQW5TP4WRGW2UNXQMMWRFWUOEC3CV7VBQ, seg_pos={0 0}, upload=463864]: Uploading piece... Dec 27 16:21:16.565203404 (528h52m16.436286449s) [noise=true , address=45.91.116.11:4118 , piece_id=TYWZQDLCTAH3UISSSRGDYZKYKSQPZWARWW6ABTWR7VPSGOJJEQYA, seg_pos={0 0}, upload=463864]: Uploading piece... Dec 27 16:21:18.274659468 (528h52m18.145742513s) [upload=463864]: Next returned nil segment Dec 27 16:21:18.274671781 (528h52m18.145754826s) [upload=463864]: Waiting for error group managing segments... Dec 27 16:21:18.550836227 (528h52m18.421919272s) [noise=true , address=145.79.227.38:4978 , piece_id=GA3LEYBS4JWYRAPIEF7TK4TIUQFJ5CITT7XLTK5FVRIK36JUIZ4Q, seg_pos={0 0}, upload=463864]: Done uploading piece. err: <nil> Dec 27 16:21:18.559524060 (528h52m18.430607095s) [noise=true , address=145.79.227.36:4518 , piece_id=EN7EVBMHHKZ3DMBBZW3NPKY64ZDFYFCERRCGOYRUJOW6LWGZVNVQ, seg_pos={0 0}, upload=463864]: Done uploading piece. err: <nil> Dec 27 16:21:18.562506048 (528h52m18.433589094s) [noise=true , address=145.79.227.42:4123 , piece_id=NCAJT4Z3MEDJ6YHV6KSKDGEKMI5W2KFTQP46YTBK2NFHZ2QIXYMA, seg_pos={0 0}, upload=463864]: Done uploading piece. err: <nil> Dec 27 16:21:18.566186567 (528h52m18.437269612s) [noise=true , address=145.79.227.16:4912 , piece_id=TYUPK4HDXYUY5KFRBLVBI3VGEOBCFOKSS52TH3QCXGUKZVCYTRYQ, seg_pos={0 0}, upload=463864]: Done uploading piece. err: <nil> Dec 27 16:21:18.569756865 (528h52m18.4408399s) [noise=true , address=145.79.227.12:4767 , piece_id=BPVIVTGUU4SI53J4Z5EAYOCM3Z2E6YLDTLXYATTCRYOMENOS34DQ, seg_pos={0 0}, upload=463864]: Done uploading piece. err: <nil> Dec 27 16:21:18.570466457 (528h52m18.441549492s) [noise=true , address=145.79.227.39:4015 , piece_id=TFLQCX6H3TM4SHWL4SFR63PSAPFC4YTK47PIKL7A765TO63AOAKA, seg_pos={0 0}, upload=463864]: Done uploading piece. err: <nil> Dec 27 16:21:18.570896956 (528h52m18.441980001s) [noise=true , address=145.79.227.51:4409 , piece_id=A27IAXSG2ECL4SS3Q4JKUOUXV4IXXKYFMMVVQMM4ZCACYQHMPVQQ, seg_pos={0 0}, upload=463864]: Done uploading piece. err: <nil> Dec 27 16:21:18.571305725 (528h52m18.44238876s) [noise=true , address=145.79.227.47:4267 , piece_id=SFQRNTSMRLPK3YGDZ6EP5DK5A66YUCJQSSYV6K7Z6VSKJ2BON52A, seg_pos={0 0}, upload=463864]: Done uploading piece. err: <nil> Dec 27 16:21:18.573260006 (528h52m18.444343051s) [noise=true , address=145.79.227.7:4295 , piece_id=UD46ASUVPMUGHRU4U66EQW5TP4WRGW2UNXQMMWRFWUOEC3CV7VBQ, seg_pos={0 0}, upload=463864]: Done uploading piece. err: <nil> Dec 27 16:21:18.574780081 (528h52m18.445863126s) [noise=true , address=145.79.227.23:4545 , piece_id=4GDHTKFHLWDZI6NOE2HY3DZFT6GHQSF3OJIDM3CLB57VC37L4B5Q, seg_pos={0 0}, upload=463864]: Done uploading piece. err: <nil> Dec 27 16:21:18.860524047 (528h52m18.731607092s) [noise=true , address=77.90.31.10:4518 , piece_id=YSPOY4DBCBURRYP2B4ZO4OFJYMEQYY6WXT4CKWIDWAFZTTHM4TTQ, seg_pos={0 0}, upload=463864]: Done uploading piece. err: <nil> Dec 27 16:21:18.923855732 (528h52m18.794938767s) [noise=true , address=45.91.116.11:4118 , piece_id=TYWZQDLCTAH3UISSSRGDYZKYKSQPZWARWW6ABTWR7VPSGOJJEQYA, seg_pos={0 0}, upload=463864]: Done uploading piece. err: <nil> Dec 27 16:21:18.984066784 (528h52m18.855149829s) [noise=true , address=45.91.116.10:4083 , piece_id=PIFBU266FRJVQNLAK6I5B3PEX2MQMHJRLYYYNEJQH2XU5FEQMW2A, seg_pos={0 0}, upload=463864]: Done uploading piece. err: <nil> Dec 27 16:21:18.984091040 (528h52m18.855174095s) [seg_pos={0 0}, upload=463864]: Segment reached optimal threshold of 13 pieces. Dec 27 16:21:18.984331538 (528h52m18.855414583s) [noise=true , address=45.91.116.4:4937 , piece_id=MGGCASLJGRK3X7Q5EAATGZEGQXYEA5HTETC245RC2QNWSVHJDJ5Q, seg_pos={0 0}, upload=463864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF Dec 27 16:21:18.984340265 (528h52m18.85542331s) [noise=true , address=45.91.116.4:4937 , piece_id=MGGCASLJGRK3X7Q5EAATGZEGQXYEA5HTETC245RC2QNWSVHJDJ5Q, seg_pos={0 0}, upload=463864]: Long tail cancellation due to optimal threshold Dec 27 16:21:18.984450224 (528h52m18.855533259s) [noise=true , address=145.79.227.43:4158 , piece_id=MY4VJBFMA3QL5P5JXSN3YY5HKNEK2FFNASJLDWWRIMOVL5POCKUA, seg_pos={0 0}, upload=463864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF Dec 27 16:21:18.984474521 (528h52m18.855557556s) [noise=true , address=145.79.227.43:4158 , piece_id=MY4VJBFMA3QL5P5JXSN3YY5HKNEK2FFNASJLDWWRIMOVL5POCKUA, seg_pos={0 0}, upload=463864]: Long tail cancellation due to optimal threshold Dec 27 16:21:18.984705952 (528h52m18.855788987s) [noise=true , address=77.90.31.21:4877 , piece_id=DFHBRNGPREJMZKZZ7WYZT6W6D4EWWYG7W2F4WK45U6GTYPADEYTA, seg_pos={0 0}, upload=463864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: context canceled; context canceled Dec 27 16:21:18.984716040 (528h52m18.855799086s) [noise=true , address=77.90.31.21:4877 , piece_id=DFHBRNGPREJMZKZZ7WYZT6W6D4EWWYG7W2F4WK45U6GTYPADEYTA, seg_pos={0 0}, upload=463864]: Long tail cancellation due to optimal threshold Dec 27 16:21:18.984763180 (528h52m18.855846225s) [seg_pos={0 0}, upload=463864]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:21:18.984872568 (528h52m18.855955603s) [upload=463864]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:21:18.984879582 (528h52m18.855962617s) [upload=463864]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:21:19.279731311 (528h52m19.150814346s) [upload=463864]: Done uploading segments... Dec 27 16:21:19.279754806 (528h52m19.150837851s) [upload=463864]: Upload finished. err: <nil> Duration: 30.002473114s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X................XX.......................................] duration: 53.629152329s trace: https://jaeger.dev.storj.io/trace/9fd07cafe8f934a FAIL (2025-12-27 16:21:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315411726 (528h16m20.186494771s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315905206 (528h16m20.186988251s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316356304 (528h16m20.187439339s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316793537 (528h16m20.187876582s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.317233315 (528h16m20.18831636s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318042255 (528h16m20.1891253s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318150922 (528h16m20.189233967s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318513712 (528h16m20.189596747s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318955464 (528h16m20.190038509s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319037570 (528h16m20.190120605s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319389541 (528h16m20.190472576s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319874033 (528h16m20.190957078s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320378844 (528h16m20.191461879s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320898604 (528h16m20.191981649s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.321434254 (528h16m20.192517289s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:21.620856000 (528h16m21.491939035s) [upload=463344]: Next returned nil segment Dec 27 15:45:21.620875698 (528h16m21.491958743s) [upload=463344]: Waiting for error group managing segments... Dec 27 15:45:21.953178480 (528h16m21.824261516s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.194632602 (528h16m22.065715647s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.195118968 (528h16m22.066202013s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.204573761 (528h16m22.075656806s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.213413343 (528h16m22.084496388s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.372654386 (528h16m22.243737431s) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.473126101 (528h16m22.344209146s) [noise=true , address=77.90.31.12:4615 , piece_id=NTYP2JTUJYSBE4NBMQHTEAY2DA333CR7V3622PVQ3OGRMSPCJK3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.484926481 (528h16m22.356009536s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.563981276 (528h16m22.435064311s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.893470090 (528h16m22.764553126s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.105921755 (528h16m22.9770048s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.153398930 (528h16m23.024481975s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.218328398 (528h16m23.089411443s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244693782 (528h16m23.115776827s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244714231 (528h16m23.115797266s) [seg_pos={0 0}, upload=463344]: Segment reached optimal threshold of 14 pieces. Dec 27 15:45:23.244907037 (528h16m23.115990073s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.244913159 (528h16m23.115996204s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245080929 (528h16m23.116163964s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245087220 (528h16m23.116170275s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245280880 (528h16m23.116363925s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection Dec 27 15:45:23.245370850 (528h16m23.116453885s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245453588 (528h16m23.116536644s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f2MspBPykwYCWAYzJ9PCAXop6mPNarJt3fa6fWZoLipZZQdy1): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245474488 (528h16m23.116557533s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245514535 (528h16m23.11659757s) [seg_pos={0 0}, upload=463344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:45:23.246027922 (528h16m23.117110977s) [upload=463344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:45:23.246044133 (528h16m23.117127168s) [upload=463344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:45:23.560988923 (528h16m23.432071968s) [upload=463344]: Done uploading segments... Dec 27 15:45:23.561009573 (528h16m23.432092618s) [upload=463344]: Upload finished. err: <nil> Duration: 53.629178259s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [.XXXX....XXXX..XX..XX......X......XX...X.............X.....X] duration: 5.001339869s trace: https://jaeger.dev.storj.io/trace/5484ae4968b2b854 FAIL (2025-12-27 16:21:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001362642s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [X.XXXX....XXXX..XX..XX......X......XX...X.............X.....] duration: 3.736678792s trace: https://jaeger.dev.storj.io/trace/6b9e6d51cd89450e FAIL (2025-12-27 16:21:08 UTC) orbiter warning: context deadline exceeded Duration: 3.736707896s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XXX...........X.......X..X.XX..X..X....XX.X....X..X.XXX..X.X] duration: 30.003812072s trace: https://jaeger.dev.storj.io/trace/3efe16af62e2decf FAIL (2025-12-27 16:20:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 16:20:13.029833410 (528h51m12.900916455s) [upload=463848]: Starting upload Dec 27 16:20:13.029967405 (528h51m12.90105046s) [upload=463848]: Done starting upload Dec 27 16:20:13.030157788 (528h51m12.901240823s) [upload=463848]: Uploading segments... Dec 27 16:20:13.030261125 (528h51m12.90134416s) [upload=463848]: Got next segment. Inline: false Dec 27 16:20:13.030302654 (528h51m12.901385699s) [upload=463848]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:20:13.030308014 (528h51m12.901391049s) [upload=463848]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:20:13.276039280 (528h51m13.147122325s) [seg_pos={0 0}, upload=463848]: Begin upload segment... Dec 27 16:20:13.276097379 (528h51m13.147180415s) [seg_pos={0 0}, upload=463848]: Done begin upload segment. Dec 27 16:20:13.276159718 (528h51m13.147242753s) [noise=true , address=77.90.31.10:4518 , piece_id=AAUD4BZD3F5WLJIUUHYIAKVXN3VBHUJOQZBDZUM3ZARQY6AQDBPA, seg_pos={0 0}, upload=463848]: Uploading piece... Dec 27 16:20:13.276168385 (528h51m13.14725143s) [noise=true , address=45.91.116.10:4082 , piece_id=IP4SNMCDVFGZPJTANO65ADOGHGRBAWQKPU22NHO2OFALCDVEBAPA, seg_pos={0 0}, upload=463848]: Uploading piece... Dec 27 16:20:13.276751786 (528h51m13.147834831s) [noise=true , address=145.79.227.23:4546 , piece_id=ZYKNTWL5ZLTLO7FVQQBYQPHE3RA53S4BYCITIQJWHBAGKMI2PX5A, seg_pos={0 0}, upload=463848]: Uploading piece... Dec 27 16:20:13.277159452 (528h51m13.148242507s) [noise=true , address=145.79.227.29:4941 , piece_id=MTPGSQ5XFRWIQ467GZY44XJ2CF73FSRQRUOXUGEZEESIA5PO5WLQ, seg_pos={0 0}, upload=463848]: Uploading piece... Dec 27 16:20:13.277567409 (528h51m13.148650455s) [noise=true , address=145.79.227.41:4086 , piece_id=WHJ7OYOWPFL5A6XSZ7TS73ZNEQ24ZBMYS7F6CLUI3REFIFCJ4Y5Q, seg_pos={0 0}, upload=463848]: Uploading piece... Dec 27 16:20:13.278290526 (528h51m13.149373561s) [noise=true , address=145.79.227.20:4438 , piece_id=MEYHZZJNVXYNCQS5Z3Q53MC3TGUWHFQTEZMV732JKDDMJU6MREBQ, seg_pos={0 0}, upload=463848]: Uploading piece... Dec 27 16:20:13.278711297 (528h51m13.149794342s) [noise=true , address=45.91.116.12:4226 , piece_id=H5ULRVK3L5WCPQJ7MDADD52KZVNZ4MQRCGDPQ4AWO2EHOWJSXCDQ, seg_pos={0 0}, upload=463848]: Uploading piece... Dec 27 16:20:13.279199997 (528h51m13.150283042s) [noise=true , address=145.79.227.38:4978 , piece_id=QGMCUUTUNKZ26CHOQYBO65ICMNEFOXIELAN3KUPZMGU4YODLD7AQ, seg_pos={0 0}, upload=463848]: Uploading piece... Dec 27 16:20:13.279656838 (528h51m13.150739873s) [noise=true , address=145.79.227.21:4474 , piece_id=Y6AGDHVQXOB5S2XARQC76DLGG3K3THL5G4KWSNK4SSFU3X67RZOA, seg_pos={0 0}, upload=463848]: Uploading piece... Dec 27 16:20:13.280086456 (528h51m13.151169491s) [noise=true , address=145.79.227.45:4231 , piece_id=UAZTHXUABYJOW6HHKKZHJFFFOETM2N6PXGLETLY35SZ3C6ZIP7PA, seg_pos={0 0}, upload=463848]: Uploading piece... Dec 27 16:20:13.280509481 (528h51m13.151592526s) [noise=true , address=45.91.116.5:4298 , piece_id=XQZT3B6FSSFV5RF3D6AHGRPMZVEERNORHNK6K7IBX47FWUZT5VPA, seg_pos={0 0}, upload=463848]: Uploading piece... Dec 27 16:20:13.280958126 (528h51m13.152041171s) [noise=true , address=145.79.227.9:4365 , piece_id=WLMJHPHNOIFWTDEMITTJQL6WQI5QXV3JRSCZU3QLXYIBMXQ4CADQ, seg_pos={0 0}, upload=463848]: Uploading piece... Dec 27 16:20:13.281389346 (528h51m13.152472431s) [noise=true , address=145.79.227.49:4339 , piece_id=XMO6674FJ7WH7VH7A4N76K7SA3LHFGJU72Z3YWJ4VZ5ULXMFHOBA, seg_pos={0 0}, upload=463848]: Uploading piece... Dec 27 16:20:13.281905058 (528h51m13.152988104s) [noise=true , address=145.79.227.19:4078 , piece_id=DSC3I4GRX6G5ZVLJX6OCXE7SB5GAPVBIFV5FL3NTAG3UJ2SSUQCQ, seg_pos={0 0}, upload=463848]: Uploading piece... Dec 27 16:20:13.292672090 (528h51m13.163755146s) [noise=true , address=145.79.227.17:4971 , piece_id=BYUTKJGNQJGAKDUSN3RX3AQO4R6VTGY3NVH7O42TCJ33TUSWDMGA, seg_pos={0 0}, upload=463848]: Uploading piece... Dec 27 16:20:13.295505296 (528h51m13.166588331s) [noise=true , address=145.79.227.42:4121 , piece_id=ZZLZX2KYETWXZNH5EQ4A23OVY6EWD45J27A5T2OSZYKHUEGP6H4Q, seg_pos={0 0}, upload=463848]: Uploading piece... Dec 27 16:20:15.007550741 (528h51m14.878633786s) [upload=463848]: Next returned nil segment Dec 27 16:20:15.007570678 (528h51m14.878653723s) [upload=463848]: Waiting for error group managing segments... Dec 27 16:20:15.301636761 (528h51m15.172719806s) [noise=true , address=145.79.227.45:4231 , piece_id=UAZTHXUABYJOW6HHKKZHJFFFOETM2N6PXGLETLY35SZ3C6ZIP7PA, seg_pos={0 0}, upload=463848]: Done uploading piece. err: <nil> Dec 27 16:20:15.302704444 (528h51m15.173787489s) [noise=true , address=145.79.227.9:4365 , piece_id=WLMJHPHNOIFWTDEMITTJQL6WQI5QXV3JRSCZU3QLXYIBMXQ4CADQ, seg_pos={0 0}, upload=463848]: Done uploading piece. err: <nil> Dec 27 16:20:15.304019499 (528h51m15.175102544s) [noise=true , address=145.79.227.21:4474 , piece_id=Y6AGDHVQXOB5S2XARQC76DLGG3K3THL5G4KWSNK4SSFU3X67RZOA, seg_pos={0 0}, upload=463848]: Done uploading piece. err: <nil> Dec 27 16:20:15.311017173 (528h51m15.182100208s) [noise=true , address=145.79.227.38:4978 , piece_id=QGMCUUTUNKZ26CHOQYBO65ICMNEFOXIELAN3KUPZMGU4YODLD7AQ, seg_pos={0 0}, upload=463848]: Done uploading piece. err: <nil> Dec 27 16:20:15.319477984 (528h51m15.190561019s) [noise=true , address=145.79.227.17:4971 , piece_id=BYUTKJGNQJGAKDUSN3RX3AQO4R6VTGY3NVH7O42TCJ33TUSWDMGA, seg_pos={0 0}, upload=463848]: Done uploading piece. err: <nil> Dec 27 16:20:15.325198256 (528h51m15.196281301s) [noise=true , address=145.79.227.42:4121 , piece_id=ZZLZX2KYETWXZNH5EQ4A23OVY6EWD45J27A5T2OSZYKHUEGP6H4Q, seg_pos={0 0}, upload=463848]: Done uploading piece. err: <nil> Dec 27 16:20:15.331246543 (528h51m15.202329589s) [noise=true , address=145.79.227.41:4086 , piece_id=WHJ7OYOWPFL5A6XSZ7TS73ZNEQ24ZBMYS7F6CLUI3REFIFCJ4Y5Q, seg_pos={0 0}, upload=463848]: Done uploading piece. err: <nil> Dec 27 16:20:15.333363905 (528h51m15.204446951s) [noise=true , address=145.79.227.23:4546 , piece_id=ZYKNTWL5ZLTLO7FVQQBYQPHE3RA53S4BYCITIQJWHBAGKMI2PX5A, seg_pos={0 0}, upload=463848]: Done uploading piece. err: <nil> Dec 27 16:20:15.338338207 (528h51m15.209421253s) [noise=true , address=145.79.227.19:4078 , piece_id=DSC3I4GRX6G5ZVLJX6OCXE7SB5GAPVBIFV5FL3NTAG3UJ2SSUQCQ, seg_pos={0 0}, upload=463848]: Done uploading piece. err: <nil> Dec 27 16:20:15.355197660 (528h51m15.226280706s) [noise=true , address=145.79.227.20:4438 , piece_id=MEYHZZJNVXYNCQS5Z3Q53MC3TGUWHFQTEZMV732JKDDMJU6MREBQ, seg_pos={0 0}, upload=463848]: Done uploading piece. err: <nil> Dec 27 16:20:15.361474983 (528h51m15.232558018s) [noise=true , address=145.79.227.29:4941 , piece_id=MTPGSQ5XFRWIQ467GZY44XJ2CF73FSRQRUOXUGEZEESIA5PO5WLQ, seg_pos={0 0}, upload=463848]: Done uploading piece. err: <nil> Dec 27 16:20:15.601164675 (528h51m15.47224771s) [noise=true , address=45.91.116.10:4082 , piece_id=IP4SNMCDVFGZPJTANO65ADOGHGRBAWQKPU22NHO2OFALCDVEBAPA, seg_pos={0 0}, upload=463848]: Done uploading piece. err: <nil> Dec 27 16:20:15.656733237 (528h51m15.527816272s) [noise=true , address=45.91.116.12:4226 , piece_id=H5ULRVK3L5WCPQJ7MDADD52KZVNZ4MQRCGDPQ4AWO2EHOWJSXCDQ, seg_pos={0 0}, upload=463848]: Done uploading piece. err: <nil> Dec 27 16:20:15.656753656 (528h51m15.527836691s) [seg_pos={0 0}, upload=463848]: Segment reached optimal threshold of 13 pieces. Dec 27 16:20:15.656989805 (528h51m15.52807286s) [noise=true , address=45.91.116.5:4298 , piece_id=XQZT3B6FSSFV5RF3D6AHGRPMZVEERNORHNK6K7IBX47FWUZT5VPA, seg_pos={0 0}, upload=463848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 27 16:20:15.656998813 (528h51m15.528081848s) [noise=true , address=45.91.116.5:4298 , piece_id=XQZT3B6FSSFV5RF3D6AHGRPMZVEERNORHNK6K7IBX47FWUZT5VPA, seg_pos={0 0}, upload=463848]: Long tail cancellation due to optimal threshold Dec 27 16:20:15.657290047 (528h51m15.528373092s) [noise=true , address=77.90.31.10:4518 , piece_id=AAUD4BZD3F5WLJIUUHYIAKVXN3VBHUJOQZBDZUM3ZARQY6AQDBPA, seg_pos={0 0}, upload=463848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF Dec 27 16:20:15.657298543 (528h51m15.528381588s) [noise=true , address=77.90.31.10:4518 , piece_id=AAUD4BZD3F5WLJIUUHYIAKVXN3VBHUJOQZBDZUM3ZARQY6AQDBPA, seg_pos={0 0}, upload=463848]: Long tail cancellation due to optimal threshold Dec 27 16:20:15.657433761 (528h51m15.528516796s) [noise=true , address=145.79.227.49:4339 , piece_id=XMO6674FJ7WH7VH7A4N76K7SA3LHFGJU72Z3YWJ4VZ5ULXMFHOBA, seg_pos={0 0}, upload=463848]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 27 16:20:15.657441265 (528h51m15.5285243s) [noise=true , address=145.79.227.49:4339 , piece_id=XMO6674FJ7WH7VH7A4N76K7SA3LHFGJU72Z3YWJ4VZ5ULXMFHOBA, seg_pos={0 0}, upload=463848]: Long tail cancellation due to optimal threshold Dec 27 16:20:15.657463428 (528h51m15.528546473s) [seg_pos={0 0}, upload=463848]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:20:15.657559200 (528h51m15.528642245s) [upload=463848]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:20:15.657566293 (528h51m15.528649349s) [upload=463848]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:20:15.964374491 (528h51m15.835457526s) [upload=463848]: Done uploading segments... Dec 27 16:20:15.964393126 (528h51m15.835476171s) [upload=463848]: Upload finished. err: <nil> Duration: 30.003837722s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [XX.XXXX....XXXX..XX..XX......X......XX...X.............X....] duration: 4.767482855s trace: https://jaeger.dev.storj.io/trace/6b9e6d51cd89450e FAIL (2025-12-27 16:20:35 UTC) orbiter warning: context deadline exceeded Duration: 4.76751233s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X................XX......................................] duration: 35.069602549s trace: https://jaeger.dev.storj.io/trace/9fd07cafe8f934a FAIL (2025-12-27 16:20:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315411726 (528h16m20.186494771s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315905206 (528h16m20.186988251s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316356304 (528h16m20.187439339s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316793537 (528h16m20.187876582s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.317233315 (528h16m20.18831636s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318042255 (528h16m20.1891253s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318150922 (528h16m20.189233967s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318513712 (528h16m20.189596747s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318955464 (528h16m20.190038509s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319037570 (528h16m20.190120605s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319389541 (528h16m20.190472576s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319874033 (528h16m20.190957078s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320378844 (528h16m20.191461879s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320898604 (528h16m20.191981649s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.321434254 (528h16m20.192517289s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:21.620856000 (528h16m21.491939035s) [upload=463344]: Next returned nil segment Dec 27 15:45:21.620875698 (528h16m21.491958743s) [upload=463344]: Waiting for error group managing segments... Dec 27 15:45:21.953178480 (528h16m21.824261516s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.194632602 (528h16m22.065715647s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.195118968 (528h16m22.066202013s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.204573761 (528h16m22.075656806s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.213413343 (528h16m22.084496388s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.372654386 (528h16m22.243737431s) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.473126101 (528h16m22.344209146s) [noise=true , address=77.90.31.12:4615 , piece_id=NTYP2JTUJYSBE4NBMQHTEAY2DA333CR7V3622PVQ3OGRMSPCJK3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.484926481 (528h16m22.356009536s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.563981276 (528h16m22.435064311s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.893470090 (528h16m22.764553126s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.105921755 (528h16m22.9770048s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.153398930 (528h16m23.024481975s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.218328398 (528h16m23.089411443s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244693782 (528h16m23.115776827s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244714231 (528h16m23.115797266s) [seg_pos={0 0}, upload=463344]: Segment reached optimal threshold of 14 pieces. Dec 27 15:45:23.244907037 (528h16m23.115990073s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.244913159 (528h16m23.115996204s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245080929 (528h16m23.116163964s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245087220 (528h16m23.116170275s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245280880 (528h16m23.116363925s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection Dec 27 15:45:23.245370850 (528h16m23.116453885s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245453588 (528h16m23.116536644s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f2MspBPykwYCWAYzJ9PCAXop6mPNarJt3fa6fWZoLipZZQdy1): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245474488 (528h16m23.116557533s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245514535 (528h16m23.11659757s) [seg_pos={0 0}, upload=463344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:45:23.246027922 (528h16m23.117110977s) [upload=463344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:45:23.246044133 (528h16m23.117127168s) [upload=463344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:45:23.560988923 (528h16m23.432071968s) [upload=463344]: Done uploading segments... Dec 27 15:45:23.561009573 (528h16m23.432092618s) [upload=463344]: Upload finished. err: <nil> Duration: 35.069626574s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.XXX...........X.......X..X.XX..X..X....XX.X....X..X.XXX..X.] duration: 5.841495042s trace: https://jaeger.dev.storj.io/trace/759fe848aee6b70 FAIL (2025-12-27 16:20:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Dec 27 16:18:55.917837053 (528h49m55.788920088s) [upload=463828]: Got next segment. Inline: false Dec 27 16:18:55.917958024 (528h49m55.789041059s) [upload=463828]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:18:55.917964075 (528h49m55.78904711s) [upload=463828]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:18:56.160213020 (528h49m56.031296055s) [seg_pos={0 0}, upload=463828]: Begin upload segment... Dec 27 16:18:56.160266291 (528h49m56.031349326s) [seg_pos={0 0}, upload=463828]: Done begin upload segment. Dec 27 16:18:56.160292972 (528h49m56.031376007s) [noise=true , address=145.79.227.17:4971 , piece_id=DFZ7JQCKQKBLAQL43OIX3HTYCMG5OBS3PP3K2T7IWNKI5YDQQAIQ, seg_pos={0 0}, upload=463828]: Uploading piece... Dec 27 16:18:56.160302951 (528h49m56.031385986s) [noise=true , address=145.79.227.28:4726 , piece_id=FXHHFOIJNIENIZ3PJX4D7FENEKWLKRM5GPAWYMNEZHJ6X26QQRKA, seg_pos={0 0}, upload=463828]: Uploading piece... Dec 27 16:18:56.160843150 (528h49m56.031926185s) [noise=true , address=145.79.227.10:4403 , piece_id=ASCHRRGRCXWQKWWNJ3HHZUELLUJY3SNDJF57Z3V727PA66OB45PQ, seg_pos={0 0}, upload=463828]: Uploading piece... Dec 27 16:18:56.161115940 (528h49m56.032198975s) [noise=true , address=145.79.227.45:4229 , piece_id=4RIJRLHIJS432BTDKXZXI5DQIVDBTIODSJPE5ZCZ53Y6VGPUI7PQ, seg_pos={0 0}, upload=463828]: Uploading piece... Dec 27 16:18:56.161407204 (528h49m56.032490239s) [noise=true , address=145.79.227.35:4483 , piece_id=HKDPAOOVHRWAMCMJZC5NLGV53DPNZD2THOM7YV45NCMWQTQHY5HQ, seg_pos={0 0}, upload=463828]: Uploading piece... Dec 27 16:18:56.162686460 (528h49m56.033769495s) [noise=true , address=45.91.116.13:4155 , piece_id=QLJ7NDGXBP5EDS7DX42QBDQA7SXGSISXB2SHG6HY3RMSCGZQVRAQ, seg_pos={0 0}, upload=463828]: Uploading piece... Dec 27 16:18:56.163136938 (528h49m56.034219973s) [noise=true , address=145.79.227.2:4114 , piece_id=GJ2SDHJORG2LQDPZKZE6DN3E6QX5AAC5A5MEJJU2H3UN7KKTG3MQ, seg_pos={0 0}, upload=463828]: Uploading piece... Dec 27 16:18:56.164203038 (528h49m56.035286073s) [noise=true , address=145.79.227.15:4876 , piece_id=KYZTFEV3Z6CDNPWYMPO5MLH522MBA3IICJD7IV2UJYBPN65AGALQ, seg_pos={0 0}, upload=463828]: Uploading piece... Dec 27 16:18:56.164403309 (528h49m56.035486354s) [noise=true , address=45.91.116.5:4300 , piece_id=KYOVFBJFWCJ2G7KS3TGXGY6CFKN7BKNLWXI7D7VCBU4FDAOYYDGQ, seg_pos={0 0}, upload=463828]: Uploading piece... Dec 27 16:18:56.165757097 (528h49m56.036840142s) [noise=true , address=145.79.227.14:4837 , piece_id=ID72YBBN3TJFOR2XJN76KYEIBXCNZVQ5OBZMTTNLNDONSYPGOJNQ, seg_pos={0 0}, upload=463828]: Uploading piece... Dec 27 16:18:56.167353316 (528h49m56.038436362s) [noise=true , address=145.79.227.51:4411 , piece_id=37OINI2FS3PXPVDMECM6C7UX54Q3KU34MHBO7IP43MGLCQKONETQ, seg_pos={0 0}, upload=463828]: Uploading piece... Dec 27 16:18:56.168646409 (528h49m56.039729444s) [noise=true , address=77.90.31.13:4627 , piece_id=UHZFCEQI4FM75RCFL7BYBOV5ZGRV26KUYPOUIU5LXCRAZ4CWZSIA, seg_pos={0 0}, upload=463828]: Uploading piece... Dec 27 16:18:56.169963607 (528h49m56.041046642s) [noise=true , address=77.90.31.21:4877 , piece_id=QRADXTSMSGRGPEBQ2DAIHDPG2IPGRJOYKU7EL45AJKZRXFBC3O5Q, seg_pos={0 0}, upload=463828]: Uploading piece... Dec 27 16:18:56.170255452 (528h49m56.041338487s) [noise=true , address=145.79.227.5:4223 , piece_id=GUDBYTSUKXLRMHKATML2MCIRFMOVGZYL7UA62PB7BAS7OMR54KZQ, seg_pos={0 0}, upload=463828]: Uploading piece... Dec 27 16:18:56.171485806 (528h49m56.042568841s) [noise=true , address=145.79.227.52:4446 , piece_id=4S4ZER6ZZ3JYYKQOEDTIGTP5TFBP2N5YR7ET5MRMJAZINMBDFJFQ, seg_pos={0 0}, upload=463828]: Uploading piece... Dec 27 16:18:56.171978664 (528h49m56.043061709s) [noise=true , address=45.91.116.8:4047 , piece_id=RL4IWJBXML373ZIW2BHOPTKHQANHEVCLXNUEK3E6AI27FVDUD4YA, seg_pos={0 0}, upload=463828]: Uploading piece... Dec 27 16:18:57.770205177 (528h49m57.641288212s) [upload=463828]: Next returned nil segment Dec 27 16:18:57.770217230 (528h49m57.641300265s) [upload=463828]: Waiting for error group managing segments... Dec 27 16:18:58.160670963 (528h49m58.031753998s) [noise=true , address=145.79.227.17:4971 , piece_id=DFZ7JQCKQKBLAQL43OIX3HTYCMG5OBS3PP3K2T7IWNKI5YDQQAIQ, seg_pos={0 0}, upload=463828]: Done uploading piece. err: <nil> Dec 27 16:18:58.162780731 (528h49m58.033863766s) [noise=true , address=145.79.227.45:4229 , piece_id=4RIJRLHIJS432BTDKXZXI5DQIVDBTIODSJPE5ZCZ53Y6VGPUI7PQ, seg_pos={0 0}, upload=463828]: Done uploading piece. err: <nil> Dec 27 16:18:58.167518302 (528h49m58.038601337s) [noise=true , address=145.79.227.14:4837 , piece_id=ID72YBBN3TJFOR2XJN76KYEIBXCNZVQ5OBZMTTNLNDONSYPGOJNQ, seg_pos={0 0}, upload=463828]: Done uploading piece. err: <nil> Dec 27 16:18:58.168086463 (528h49m58.039169498s) [noise=true , address=145.79.227.10:4403 , piece_id=ASCHRRGRCXWQKWWNJ3HHZUELLUJY3SNDJF57Z3V727PA66OB45PQ, seg_pos={0 0}, upload=463828]: Done uploading piece. err: <nil> Dec 27 16:18:58.169353116 (528h49m58.040436151s) [noise=true , address=145.79.227.2:4114 , piece_id=GJ2SDHJORG2LQDPZKZE6DN3E6QX5AAC5A5MEJJU2H3UN7KKTG3MQ, seg_pos={0 0}, upload=463828]: Done uploading piece. err: <nil> Dec 27 16:18:58.171024669 (528h49m58.042107704s) [noise=true , address=145.79.227.15:4876 , piece_id=KYZTFEV3Z6CDNPWYMPO5MLH522MBA3IICJD7IV2UJYBPN65AGALQ, seg_pos={0 0}, upload=463828]: Done uploading piece. err: <nil> Dec 27 16:18:58.172056533 (528h49m58.043139578s) [noise=true , address=145.79.227.28:4726 , piece_id=FXHHFOIJNIENIZ3PJX4D7FENEKWLKRM5GPAWYMNEZHJ6X26QQRKA, seg_pos={0 0}, upload=463828]: Done uploading piece. err: <nil> Dec 27 16:18:58.176365759 (528h49m58.047448794s) [noise=true , address=145.79.227.52:4446 , piece_id=4S4ZER6ZZ3JYYKQOEDTIGTP5TFBP2N5YR7ET5MRMJAZINMBDFJFQ, seg_pos={0 0}, upload=463828]: Done uploading piece. err: <nil> Dec 27 16:18:58.182290791 (528h49m58.053373836s) [noise=true , address=145.79.227.5:4223 , piece_id=GUDBYTSUKXLRMHKATML2MCIRFMOVGZYL7UA62PB7BAS7OMR54KZQ, seg_pos={0 0}, upload=463828]: Done uploading piece. err: <nil> Dec 27 16:18:58.187034624 (528h49m58.058117669s) [noise=true , address=145.79.227.51:4411 , piece_id=37OINI2FS3PXPVDMECM6C7UX54Q3KU34MHBO7IP43MGLCQKONETQ, seg_pos={0 0}, upload=463828]: Done uploading piece. err: <nil> Dec 27 16:18:58.554777802 (528h49m58.425860847s) [noise=true , address=45.91.116.13:4155 , piece_id=QLJ7NDGXBP5EDS7DX42QBDQA7SXGSISXB2SHG6HY3RMSCGZQVRAQ, seg_pos={0 0}, upload=463828]: Done uploading piece. err: <nil> Dec 27 16:18:58.724125489 (528h49m58.595208524s) [noise=true , address=77.90.31.21:4877 , piece_id=QRADXTSMSGRGPEBQ2DAIHDPG2IPGRJOYKU7EL45AJKZRXFBC3O5Q, seg_pos={0 0}, upload=463828]: Done uploading piece. err: <nil> Dec 27 16:18:59.147772928 (528h49m59.018855973s) [noise=true , address=45.91.116.5:4300 , piece_id=KYOVFBJFWCJ2G7KS3TGXGY6CFKN7BKNLWXI7D7VCBU4FDAOYYDGQ, seg_pos={0 0}, upload=463828]: Done uploading piece. err: <nil> Dec 27 16:18:59.147795511 (528h49m59.018878546s) [seg_pos={0 0}, upload=463828]: Segment reached optimal threshold of 13 pieces. Dec 27 16:18:59.147994350 (528h49m59.019077385s) [noise=true , address=145.79.227.35:4483 , piece_id=HKDPAOOVHRWAMCMJZC5NLGV53DPNZD2THOM7YV45NCMWQTQHY5HQ, seg_pos={0 0}, upload=463828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 27 16:18:59.148002135 (528h49m59.01908518s) [noise=true , address=145.79.227.35:4483 , piece_id=HKDPAOOVHRWAMCMJZC5NLGV53DPNZD2THOM7YV45NCMWQTQHY5HQ, seg_pos={0 0}, upload=463828]: Long tail cancellation due to optimal threshold Dec 27 16:18:59.148167619 (528h49m59.019250664s) [noise=true , address=45.91.116.8:4047 , piece_id=RL4IWJBXML373ZIW2BHOPTKHQANHEVCLXNUEK3E6AI27FVDUD4YA, seg_pos={0 0}, upload=463828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:52174->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:52174->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52174->45.91.116.8:4047: use of closed network connection Dec 27 16:18:59.148174623 (528h49m59.019257669s) [noise=true , address=45.91.116.8:4047 , piece_id=RL4IWJBXML373ZIW2BHOPTKHQANHEVCLXNUEK3E6AI27FVDUD4YA, seg_pos={0 0}, upload=463828]: Long tail cancellation due to optimal threshold Dec 27 16:18:59.148251450 (528h49m59.019334485s) [noise=true , address=77.90.31.13:4627 , piece_id=UHZFCEQI4FM75RCFL7BYBOV5ZGRV26KUYPOUIU5LXCRAZ4CWZSIA, seg_pos={0 0}, upload=463828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:43818->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:43818->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43818->77.90.31.13:4627: use of closed network connection Dec 27 16:18:59.148267179 (528h49m59.019350214s) [noise=true , address=77.90.31.13:4627 , piece_id=UHZFCEQI4FM75RCFL7BYBOV5ZGRV26KUYPOUIU5LXCRAZ4CWZSIA, seg_pos={0 0}, upload=463828]: Long tail cancellation due to optimal threshold Dec 27 16:18:59.148291836 (528h49m59.019374871s) [seg_pos={0 0}, upload=463828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:18:59.148399611 (528h49m59.019482646s) [upload=463828]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:18:59.148405773 (528h49m59.019488808s) [upload=463828]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:18:59.431938930 (528h49m59.303021975s) [upload=463828]: Done uploading segments... Dec 27 16:18:59.431959349 (528h49m59.303042384s) [upload=463828]: Upload finished. err: <nil> Duration: 5.841525951s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X................XX.....................................] duration: 43.219017683s trace: https://jaeger.dev.storj.io/trace/9fd07cafe8f934a FAIL (2025-12-27 16:19:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315411726 (528h16m20.186494771s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315905206 (528h16m20.186988251s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316356304 (528h16m20.187439339s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316793537 (528h16m20.187876582s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.317233315 (528h16m20.18831636s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318042255 (528h16m20.1891253s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318150922 (528h16m20.189233967s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318513712 (528h16m20.189596747s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318955464 (528h16m20.190038509s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319037570 (528h16m20.190120605s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319389541 (528h16m20.190472576s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319874033 (528h16m20.190957078s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320378844 (528h16m20.191461879s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320898604 (528h16m20.191981649s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.321434254 (528h16m20.192517289s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:21.620856000 (528h16m21.491939035s) [upload=463344]: Next returned nil segment Dec 27 15:45:21.620875698 (528h16m21.491958743s) [upload=463344]: Waiting for error group managing segments... Dec 27 15:45:21.953178480 (528h16m21.824261516s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.194632602 (528h16m22.065715647s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.195118968 (528h16m22.066202013s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.204573761 (528h16m22.075656806s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.213413343 (528h16m22.084496388s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.372654386 (528h16m22.243737431s) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.473126101 (528h16m22.344209146s) [noise=true , address=77.90.31.12:4615 , piece_id=NTYP2JTUJYSBE4NBMQHTEAY2DA333CR7V3622PVQ3OGRMSPCJK3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.484926481 (528h16m22.356009536s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.563981276 (528h16m22.435064311s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.893470090 (528h16m22.764553126s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.105921755 (528h16m22.9770048s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.153398930 (528h16m23.024481975s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.218328398 (528h16m23.089411443s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244693782 (528h16m23.115776827s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244714231 (528h16m23.115797266s) [seg_pos={0 0}, upload=463344]: Segment reached optimal threshold of 14 pieces. Dec 27 15:45:23.244907037 (528h16m23.115990073s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.244913159 (528h16m23.115996204s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245080929 (528h16m23.116163964s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245087220 (528h16m23.116170275s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245280880 (528h16m23.116363925s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection Dec 27 15:45:23.245370850 (528h16m23.116453885s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245453588 (528h16m23.116536644s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f2MspBPykwYCWAYzJ9PCAXop6mPNarJt3fa6fWZoLipZZQdy1): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245474488 (528h16m23.116557533s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245514535 (528h16m23.11659757s) [seg_pos={0 0}, upload=463344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:45:23.246027922 (528h16m23.117110977s) [upload=463344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:45:23.246044133 (528h16m23.117127168s) [upload=463344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:45:23.560988923 (528h16m23.432071968s) [upload=463344]: Done uploading segments... Dec 27 15:45:23.561009573 (528h16m23.432092618s) [upload=463344]: Upload finished. err: <nil> Duration: 43.219039435s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [.XX.XXXX....XXXX..XX..XX......X......XX...X.............X...] duration: 3.512052174s trace: https://jaeger.dev.storj.io/trace/6b9e6d51cd89450e FAIL (2025-12-27 16:19:43 UTC) orbiter warning: context deadline exceeded Duration: 3.512079225s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..XXX...........X.......X..X.XX..X..X....XX.X....X..X.XXX..X] duration: 30.003203434s trace: https://jaeger.dev.storj.io/trace/759fe848aee6b70 FAIL (2025-12-27 16:19:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== egments... Dec 27 16:18:55.917837053 (528h49m55.788920088s) [upload=463828]: Got next segment. Inline: false Dec 27 16:18:55.917958024 (528h49m55.789041059s) [upload=463828]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:18:55.917964075 (528h49m55.78904711s) [upload=463828]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:18:56.160213020 (528h49m56.031296055s) [seg_pos={0 0}, upload=463828]: Begin upload segment... Dec 27 16:18:56.160266291 (528h49m56.031349326s) [seg_pos={0 0}, upload=463828]: Done begin upload segment. Dec 27 16:18:56.160292972 (528h49m56.031376007s) [noise=true , address=145.79.227.17:4971 , piece_id=DFZ7JQCKQKBLAQL43OIX3HTYCMG5OBS3PP3K2T7IWNKI5YDQQAIQ, seg_pos={0 0}, upload=463828]: Uploading piece... Dec 27 16:18:56.160302951 (528h49m56.031385986s) [noise=true , address=145.79.227.28:4726 , piece_id=FXHHFOIJNIENIZ3PJX4D7FENEKWLKRM5GPAWYMNEZHJ6X26QQRKA, seg_pos={0 0}, upload=463828]: Uploading piece... Dec 27 16:18:56.160843150 (528h49m56.031926185s) [noise=true , address=145.79.227.10:4403 , piece_id=ASCHRRGRCXWQKWWNJ3HHZUELLUJY3SNDJF57Z3V727PA66OB45PQ, seg_pos={0 0}, upload=463828]: Uploading piece... Dec 27 16:18:56.161115940 (528h49m56.032198975s) [noise=true , address=145.79.227.45:4229 , piece_id=4RIJRLHIJS432BTDKXZXI5DQIVDBTIODSJPE5ZCZ53Y6VGPUI7PQ, seg_pos={0 0}, upload=463828]: Uploading piece... Dec 27 16:18:56.161407204 (528h49m56.032490239s) [noise=true , address=145.79.227.35:4483 , piece_id=HKDPAOOVHRWAMCMJZC5NLGV53DPNZD2THOM7YV45NCMWQTQHY5HQ, seg_pos={0 0}, upload=463828]: Uploading piece... Dec 27 16:18:56.162686460 (528h49m56.033769495s) [noise=true , address=45.91.116.13:4155 , piece_id=QLJ7NDGXBP5EDS7DX42QBDQA7SXGSISXB2SHG6HY3RMSCGZQVRAQ, seg_pos={0 0}, upload=463828]: Uploading piece... Dec 27 16:18:56.163136938 (528h49m56.034219973s) [noise=true , address=145.79.227.2:4114 , piece_id=GJ2SDHJORG2LQDPZKZE6DN3E6QX5AAC5A5MEJJU2H3UN7KKTG3MQ, seg_pos={0 0}, upload=463828]: Uploading piece... Dec 27 16:18:56.164203038 (528h49m56.035286073s) [noise=true , address=145.79.227.15:4876 , piece_id=KYZTFEV3Z6CDNPWYMPO5MLH522MBA3IICJD7IV2UJYBPN65AGALQ, seg_pos={0 0}, upload=463828]: Uploading piece... Dec 27 16:18:56.164403309 (528h49m56.035486354s) [noise=true , address=45.91.116.5:4300 , piece_id=KYOVFBJFWCJ2G7KS3TGXGY6CFKN7BKNLWXI7D7VCBU4FDAOYYDGQ, seg_pos={0 0}, upload=463828]: Uploading piece... Dec 27 16:18:56.165757097 (528h49m56.036840142s) [noise=true , address=145.79.227.14:4837 , piece_id=ID72YBBN3TJFOR2XJN76KYEIBXCNZVQ5OBZMTTNLNDONSYPGOJNQ, seg_pos={0 0}, upload=463828]: Uploading piece... Dec 27 16:18:56.167353316 (528h49m56.038436362s) [noise=true , address=145.79.227.51:4411 , piece_id=37OINI2FS3PXPVDMECM6C7UX54Q3KU34MHBO7IP43MGLCQKONETQ, seg_pos={0 0}, upload=463828]: Uploading piece... Dec 27 16:18:56.168646409 (528h49m56.039729444s) [noise=true , address=77.90.31.13:4627 , piece_id=UHZFCEQI4FM75RCFL7BYBOV5ZGRV26KUYPOUIU5LXCRAZ4CWZSIA, seg_pos={0 0}, upload=463828]: Uploading piece... Dec 27 16:18:56.169963607 (528h49m56.041046642s) [noise=true , address=77.90.31.21:4877 , piece_id=QRADXTSMSGRGPEBQ2DAIHDPG2IPGRJOYKU7EL45AJKZRXFBC3O5Q, seg_pos={0 0}, upload=463828]: Uploading piece... Dec 27 16:18:56.170255452 (528h49m56.041338487s) [noise=true , address=145.79.227.5:4223 , piece_id=GUDBYTSUKXLRMHKATML2MCIRFMOVGZYL7UA62PB7BAS7OMR54KZQ, seg_pos={0 0}, upload=463828]: Uploading piece... Dec 27 16:18:56.171485806 (528h49m56.042568841s) [noise=true , address=145.79.227.52:4446 , piece_id=4S4ZER6ZZ3JYYKQOEDTIGTP5TFBP2N5YR7ET5MRMJAZINMBDFJFQ, seg_pos={0 0}, upload=463828]: Uploading piece... Dec 27 16:18:56.171978664 (528h49m56.043061709s) [noise=true , address=45.91.116.8:4047 , piece_id=RL4IWJBXML373ZIW2BHOPTKHQANHEVCLXNUEK3E6AI27FVDUD4YA, seg_pos={0 0}, upload=463828]: Uploading piece... Dec 27 16:18:57.770205177 (528h49m57.641288212s) [upload=463828]: Next returned nil segment Dec 27 16:18:57.770217230 (528h49m57.641300265s) [upload=463828]: Waiting for error group managing segments... Dec 27 16:18:58.160670963 (528h49m58.031753998s) [noise=true , address=145.79.227.17:4971 , piece_id=DFZ7JQCKQKBLAQL43OIX3HTYCMG5OBS3PP3K2T7IWNKI5YDQQAIQ, seg_pos={0 0}, upload=463828]: Done uploading piece. err: <nil> Dec 27 16:18:58.162780731 (528h49m58.033863766s) [noise=true , address=145.79.227.45:4229 , piece_id=4RIJRLHIJS432BTDKXZXI5DQIVDBTIODSJPE5ZCZ53Y6VGPUI7PQ, seg_pos={0 0}, upload=463828]: Done uploading piece. err: <nil> Dec 27 16:18:58.167518302 (528h49m58.038601337s) [noise=true , address=145.79.227.14:4837 , piece_id=ID72YBBN3TJFOR2XJN76KYEIBXCNZVQ5OBZMTTNLNDONSYPGOJNQ, seg_pos={0 0}, upload=463828]: Done uploading piece. err: <nil> Dec 27 16:18:58.168086463 (528h49m58.039169498s) [noise=true , address=145.79.227.10:4403 , piece_id=ASCHRRGRCXWQKWWNJ3HHZUELLUJY3SNDJF57Z3V727PA66OB45PQ, seg_pos={0 0}, upload=463828]: Done uploading piece. err: <nil> Dec 27 16:18:58.169353116 (528h49m58.040436151s) [noise=true , address=145.79.227.2:4114 , piece_id=GJ2SDHJORG2LQDPZKZE6DN3E6QX5AAC5A5MEJJU2H3UN7KKTG3MQ, seg_pos={0 0}, upload=463828]: Done uploading piece. err: <nil> Dec 27 16:18:58.171024669 (528h49m58.042107704s) [noise=true , address=145.79.227.15:4876 , piece_id=KYZTFEV3Z6CDNPWYMPO5MLH522MBA3IICJD7IV2UJYBPN65AGALQ, seg_pos={0 0}, upload=463828]: Done uploading piece. err: <nil> Dec 27 16:18:58.172056533 (528h49m58.043139578s) [noise=true , address=145.79.227.28:4726 , piece_id=FXHHFOIJNIENIZ3PJX4D7FENEKWLKRM5GPAWYMNEZHJ6X26QQRKA, seg_pos={0 0}, upload=463828]: Done uploading piece. err: <nil> Dec 27 16:18:58.176365759 (528h49m58.047448794s) [noise=true , address=145.79.227.52:4446 , piece_id=4S4ZER6ZZ3JYYKQOEDTIGTP5TFBP2N5YR7ET5MRMJAZINMBDFJFQ, seg_pos={0 0}, upload=463828]: Done uploading piece. err: <nil> Dec 27 16:18:58.182290791 (528h49m58.053373836s) [noise=true , address=145.79.227.5:4223 , piece_id=GUDBYTSUKXLRMHKATML2MCIRFMOVGZYL7UA62PB7BAS7OMR54KZQ, seg_pos={0 0}, upload=463828]: Done uploading piece. err: <nil> Dec 27 16:18:58.187034624 (528h49m58.058117669s) [noise=true , address=145.79.227.51:4411 , piece_id=37OINI2FS3PXPVDMECM6C7UX54Q3KU34MHBO7IP43MGLCQKONETQ, seg_pos={0 0}, upload=463828]: Done uploading piece. err: <nil> Dec 27 16:18:58.554777802 (528h49m58.425860847s) [noise=true , address=45.91.116.13:4155 , piece_id=QLJ7NDGXBP5EDS7DX42QBDQA7SXGSISXB2SHG6HY3RMSCGZQVRAQ, seg_pos={0 0}, upload=463828]: Done uploading piece. err: <nil> Dec 27 16:18:58.724125489 (528h49m58.595208524s) [noise=true , address=77.90.31.21:4877 , piece_id=QRADXTSMSGRGPEBQ2DAIHDPG2IPGRJOYKU7EL45AJKZRXFBC3O5Q, seg_pos={0 0}, upload=463828]: Done uploading piece. err: <nil> Dec 27 16:18:59.147772928 (528h49m59.018855973s) [noise=true , address=45.91.116.5:4300 , piece_id=KYOVFBJFWCJ2G7KS3TGXGY6CFKN7BKNLWXI7D7VCBU4FDAOYYDGQ, seg_pos={0 0}, upload=463828]: Done uploading piece. err: <nil> Dec 27 16:18:59.147795511 (528h49m59.018878546s) [seg_pos={0 0}, upload=463828]: Segment reached optimal threshold of 13 pieces. Dec 27 16:18:59.147994350 (528h49m59.019077385s) [noise=true , address=145.79.227.35:4483 , piece_id=HKDPAOOVHRWAMCMJZC5NLGV53DPNZD2THOM7YV45NCMWQTQHY5HQ, seg_pos={0 0}, upload=463828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 27 16:18:59.148002135 (528h49m59.01908518s) [noise=true , address=145.79.227.35:4483 , piece_id=HKDPAOOVHRWAMCMJZC5NLGV53DPNZD2THOM7YV45NCMWQTQHY5HQ, seg_pos={0 0}, upload=463828]: Long tail cancellation due to optimal threshold Dec 27 16:18:59.148167619 (528h49m59.019250664s) [noise=true , address=45.91.116.8:4047 , piece_id=RL4IWJBXML373ZIW2BHOPTKHQANHEVCLXNUEK3E6AI27FVDUD4YA, seg_pos={0 0}, upload=463828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:52174->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:52174->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52174->45.91.116.8:4047: use of closed network connection Dec 27 16:18:59.148174623 (528h49m59.019257669s) [noise=true , address=45.91.116.8:4047 , piece_id=RL4IWJBXML373ZIW2BHOPTKHQANHEVCLXNUEK3E6AI27FVDUD4YA, seg_pos={0 0}, upload=463828]: Long tail cancellation due to optimal threshold Dec 27 16:18:59.148251450 (528h49m59.019334485s) [noise=true , address=77.90.31.13:4627 , piece_id=UHZFCEQI4FM75RCFL7BYBOV5ZGRV26KUYPOUIU5LXCRAZ4CWZSIA, seg_pos={0 0}, upload=463828]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:43818->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:43818->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43818->77.90.31.13:4627: use of closed network connection Dec 27 16:18:59.148267179 (528h49m59.019350214s) [noise=true , address=77.90.31.13:4627 , piece_id=UHZFCEQI4FM75RCFL7BYBOV5ZGRV26KUYPOUIU5LXCRAZ4CWZSIA, seg_pos={0 0}, upload=463828]: Long tail cancellation due to optimal threshold Dec 27 16:18:59.148291836 (528h49m59.019374871s) [seg_pos={0 0}, upload=463828]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:18:59.148399611 (528h49m59.019482646s) [upload=463828]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:18:59.148405773 (528h49m59.019488808s) [upload=463828]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:18:59.431938930 (528h49m59.303021975s) [upload=463828]: Done uploading segments... Dec 27 16:18:59.431959349 (528h49m59.303042384s) [upload=463828]: Upload finished. err: <nil> Duration: 30.00322745s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [..XX.XXXX....XXXX..XX..XX......X......XX...X.............X..] duration: 3.475416233s trace: https://jaeger.dev.storj.io/trace/6b9e6d51cd89450e FAIL (2025-12-27 16:19:04 UTC) orbiter warning: context deadline exceeded Duration: 3.475444067s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X................XX....................................] duration: 46.478225026s trace: https://jaeger.dev.storj.io/trace/9fd07cafe8f934a FAIL (2025-12-27 16:18:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315411726 (528h16m20.186494771s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315905206 (528h16m20.186988251s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316356304 (528h16m20.187439339s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316793537 (528h16m20.187876582s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.317233315 (528h16m20.18831636s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318042255 (528h16m20.1891253s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318150922 (528h16m20.189233967s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318513712 (528h16m20.189596747s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318955464 (528h16m20.190038509s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319037570 (528h16m20.190120605s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319389541 (528h16m20.190472576s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319874033 (528h16m20.190957078s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320378844 (528h16m20.191461879s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320898604 (528h16m20.191981649s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.321434254 (528h16m20.192517289s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:21.620856000 (528h16m21.491939035s) [upload=463344]: Next returned nil segment Dec 27 15:45:21.620875698 (528h16m21.491958743s) [upload=463344]: Waiting for error group managing segments... Dec 27 15:45:21.953178480 (528h16m21.824261516s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.194632602 (528h16m22.065715647s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.195118968 (528h16m22.066202013s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.204573761 (528h16m22.075656806s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.213413343 (528h16m22.084496388s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.372654386 (528h16m22.243737431s) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.473126101 (528h16m22.344209146s) [noise=true , address=77.90.31.12:4615 , piece_id=NTYP2JTUJYSBE4NBMQHTEAY2DA333CR7V3622PVQ3OGRMSPCJK3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.484926481 (528h16m22.356009536s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.563981276 (528h16m22.435064311s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.893470090 (528h16m22.764553126s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.105921755 (528h16m22.9770048s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.153398930 (528h16m23.024481975s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.218328398 (528h16m23.089411443s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244693782 (528h16m23.115776827s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244714231 (528h16m23.115797266s) [seg_pos={0 0}, upload=463344]: Segment reached optimal threshold of 14 pieces. Dec 27 15:45:23.244907037 (528h16m23.115990073s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.244913159 (528h16m23.115996204s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245080929 (528h16m23.116163964s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245087220 (528h16m23.116170275s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245280880 (528h16m23.116363925s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection Dec 27 15:45:23.245370850 (528h16m23.116453885s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245453588 (528h16m23.116536644s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f2MspBPykwYCWAYzJ9PCAXop6mPNarJt3fa6fWZoLipZZQdy1): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245474488 (528h16m23.116557533s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245514535 (528h16m23.11659757s) [seg_pos={0 0}, upload=463344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:45:23.246027922 (528h16m23.117110977s) [upload=463344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:45:23.246044133 (528h16m23.117127168s) [upload=463344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:45:23.560988923 (528h16m23.432071968s) [upload=463344]: Done uploading segments... Dec 27 15:45:23.561009573 (528h16m23.432092618s) [upload=463344]: Upload finished. err: <nil> Duration: 46.478244612s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X..XXX...........X.......X..X.XX..X..X....XX.X....X..X.XXX..] duration: 5.83989786s trace: https://jaeger.dev.storj.io/trace/6d6e199525efdebc FAIL (2025-12-27 16:18:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 16:17:04.059444495 (528h48m3.93052755s) [upload=463799]: Starting upload Dec 27 16:17:04.059548012 (528h48m3.930631057s) [upload=463799]: Done starting upload Dec 27 16:17:04.059614218 (528h48m3.930697263s) [upload=463799]: Uploading segments... Dec 27 16:17:04.059805102 (528h48m3.930888147s) [upload=463799]: Got next segment. Inline: false Dec 27 16:17:04.059854916 (528h48m3.930937961s) [upload=463799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:17:04.059861759 (528h48m3.930944814s) [upload=463799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:17:04.303537741 (528h48m4.174620806s) [seg_pos={0 0}, upload=463799]: Begin upload segment... Dec 27 16:17:04.303607855 (528h48m4.17469091s) [seg_pos={0 0}, upload=463799]: Done begin upload segment. Dec 27 16:17:04.303637271 (528h48m4.174720316s) [noise=true , address=145.79.227.15:4874 , piece_id=NOL5V5JW34ZHFLUTMX2YCPW2R5ONUXLZH6IIFC3TFPSKAB56JGCQ, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.303652860 (528h48m4.174735905s) [noise=true , address=145.79.227.21:4474 , piece_id=OSOWWSVI2H4S5Y2D3SYX6GYSVYD3CWTOTJ56V4DGM7KDTH7LMODQ, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.304269284 (528h48m4.175352349s) [noise=true , address=45.91.116.7:4012 , piece_id=YQPGCR5UWAHNPU34H2LVJJXUJGRDJAJSFMH7PWUPOIIKBFUPDVCA, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.304868345 (528h48m4.17595142s) [noise=true , address=145.79.227.5:4223 , piece_id=OXYRSSWS5K354GMTIMYLF2CV4WBAAVCUBMECMCZP4OK7VTEGKN3A, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.305382454 (528h48m4.176465499s) [noise=true , address=145.79.227.8:4329 , piece_id=HNCYY223DBE3DG2VVKGEUURAONICIXDI32MVYWLI3IHBNHJDZR4Q, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.305657157 (528h48m4.176740202s) [noise=true , address=145.79.227.3:4149 , piece_id=WJCBBVQKT7LDGHTPQ7KZNIBMCOR7VHKCL3HN23WK4ISWDS4U5I7A, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.306516173 (528h48m4.177599218s) [noise=true , address=145.79.227.48:4302 , piece_id=3VZHUGCL5NSNDA3RTZDK6DNPTSBUOANENH5PEAFROZKQRTJ6ZAZQ, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.306981419 (528h48m4.178064464s) [noise=true , address=145.79.227.34:4907 , piece_id=2SWKNXYL4DSQGKDLRSCUBWWCAS7UOHPYM6OYQ3MX7FGVK5UWDZIA, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.307495368 (528h48m4.178578423s) [noise=true , address=145.79.227.45:4231 , piece_id=HEWZ3DKOG4ZJDEFUEQYHEWK5KUIRHSQC4RSVVST7PZJZS5LO2PUQ, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.307992053 (528h48m4.179075098s) [noise=true , address=145.79.227.50:4374 , piece_id=ZQZYXIDCD4FCPAVRINDZL6ADZZBMQUAGZIJA7GQTQRJJM5RFJLEA, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.308595772 (528h48m4.179678818s) [noise=true , address=145.79.227.51:4411 , piece_id=OUWBINBHJPG73DYCI56DU3Z6DJTIJRVXOLAONQFQG45J7HNMM2FA, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.309183612 (528h48m4.180266657s) [noise=true , address=145.79.227.27:4691 , piece_id=CPZC2ZSPWYVXEZO3BOIYYW5Z4REXJBZTUXCB62GK5OWLBBFDHVQQ, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.309783004 (528h48m4.180866049s) [noise=true , address=45.91.116.8:4046 , piece_id=FEHWPJYHTNY3MUGTTOCBHCPOG4QERTSX24BNQA2KE3DBUFZQ6K4Q, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.312066582 (528h48m4.183149627s) [noise=true , address=145.79.227.37:4565 , piece_id=5G5WHRDU7JJS2NBXOEGKVJVUN3ASOQ5NBPEURWY6BNGTFSDMTEHQ, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.314315424 (528h48m4.185398469s) [noise=true , address=77.90.31.18:4808 , piece_id=5CO5LPIO5W653525HP6OW3E2TPDAKUEMDPUJSM2OS7NUVXTZ5MCA, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.316040609 (528h48m4.187123644s) [noise=true , address=77.90.31.17:4773 , piece_id=64D4YBFA4FPJR6EOTLSEKQBNZIS36LY3GJHXMJ4IWJSJPY3GBFSA, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:06.023052108 (528h48m5.894135154s) [upload=463799]: Next returned nil segment Dec 27 16:17:06.023075864 (528h48m5.894158909s) [upload=463799]: Waiting for error group managing segments... Dec 27 16:17:06.299063353 (528h48m6.170146388s) [noise=true , address=145.79.227.15:4874 , piece_id=NOL5V5JW34ZHFLUTMX2YCPW2R5ONUXLZH6IIFC3TFPSKAB56JGCQ, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.306970299 (528h48m6.178053334s) [noise=true , address=145.79.227.21:4474 , piece_id=OSOWWSVI2H4S5Y2D3SYX6GYSVYD3CWTOTJ56V4DGM7KDTH7LMODQ, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.313053092 (528h48m6.184136127s) [noise=true , address=145.79.227.5:4223 , piece_id=OXYRSSWS5K354GMTIMYLF2CV4WBAAVCUBMECMCZP4OK7VTEGKN3A, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.327281685 (528h48m6.19836472s) [noise=true , address=145.79.227.8:4329 , piece_id=HNCYY223DBE3DG2VVKGEUURAONICIXDI32MVYWLI3IHBNHJDZR4Q, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.329546667 (528h48m6.200629702s) [noise=true , address=145.79.227.27:4691 , piece_id=CPZC2ZSPWYVXEZO3BOIYYW5Z4REXJBZTUXCB62GK5OWLBBFDHVQQ, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.332383259 (528h48m6.203466294s) [noise=true , address=145.79.227.3:4149 , piece_id=WJCBBVQKT7LDGHTPQ7KZNIBMCOR7VHKCL3HN23WK4ISWDS4U5I7A, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.333616738 (528h48m6.204699783s) [noise=true , address=145.79.227.45:4231 , piece_id=HEWZ3DKOG4ZJDEFUEQYHEWK5KUIRHSQC4RSVVST7PZJZS5LO2PUQ, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.337131420 (528h48m6.208214455s) [noise=true , address=145.79.227.37:4565 , piece_id=5G5WHRDU7JJS2NBXOEGKVJVUN3ASOQ5NBPEURWY6BNGTFSDMTEHQ, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.338729493 (528h48m6.209812528s) [noise=true , address=145.79.227.51:4411 , piece_id=OUWBINBHJPG73DYCI56DU3Z6DJTIJRVXOLAONQFQG45J7HNMM2FA, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.356759526 (528h48m6.227842561s) [noise=true , address=145.79.227.50:4374 , piece_id=ZQZYXIDCD4FCPAVRINDZL6ADZZBMQUAGZIJA7GQTQRJJM5RFJLEA, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.360719556 (528h48m6.231802592s) [noise=true , address=145.79.227.48:4302 , piece_id=3VZHUGCL5NSNDA3RTZDK6DNPTSBUOANENH5PEAFROZKQRTJ6ZAZQ, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.572641544 (528h48m6.443724599s) [noise=true , address=45.91.116.7:4012 , piece_id=YQPGCR5UWAHNPU34H2LVJJXUJGRDJAJSFMH7PWUPOIIKBFUPDVCA, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.588753143 (528h48m6.459836198s) [noise=true , address=45.91.116.8:4046 , piece_id=FEHWPJYHTNY3MUGTTOCBHCPOG4QERTSX24BNQA2KE3DBUFZQ6K4Q, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.588779943 (528h48m6.459862988s) [seg_pos={0 0}, upload=463799]: Segment reached optimal threshold of 13 pieces. Dec 27 16:17:06.588995193 (528h48m6.460078238s) [noise=true , address=77.90.31.18:4808 , piece_id=5CO5LPIO5W653525HP6OW3E2TPDAKUEMDPUJSM2OS7NUVXTZ5MCA, seg_pos={0 0}, upload=463799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 27 16:17:06.589002949 (528h48m6.460086004s) [noise=true , address=77.90.31.18:4808 , piece_id=5CO5LPIO5W653525HP6OW3E2TPDAKUEMDPUJSM2OS7NUVXTZ5MCA, seg_pos={0 0}, upload=463799]: Long tail cancellation due to optimal threshold Dec 27 16:17:06.589334710 (528h48m6.460417755s) [noise=true , address=145.79.227.34:4907 , piece_id=2SWKNXYL4DSQGKDLRSCUBWWCAS7UOHPYM6OYQ3MX7FGVK5UWDZIA, seg_pos={0 0}, upload=463799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 27 16:17:06.589343436 (528h48m6.460426481s) [noise=true , address=145.79.227.34:4907 , piece_id=2SWKNXYL4DSQGKDLRSCUBWWCAS7UOHPYM6OYQ3MX7FGVK5UWDZIA, seg_pos={0 0}, upload=463799]: Long tail cancellation due to optimal threshold Dec 27 16:17:06.589498211 (528h48m6.460581257s) [noise=true , address=77.90.31.17:4773 , piece_id=64D4YBFA4FPJR6EOTLSEKQBNZIS36LY3GJHXMJ4IWJSJPY3GBFSA, seg_pos={0 0}, upload=463799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:56700->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:56700->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56700->77.90.31.17:4773: use of closed network connection Dec 27 16:17:06.589507589 (528h48m6.460590644s) [noise=true , address=77.90.31.17:4773 , piece_id=64D4YBFA4FPJR6EOTLSEKQBNZIS36LY3GJHXMJ4IWJSJPY3GBFSA, seg_pos={0 0}, upload=463799]: Long tail cancellation due to optimal threshold Dec 27 16:17:06.589525594 (528h48m6.460608649s) [seg_pos={0 0}, upload=463799]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:17:06.589605305 (528h48m6.46068836s) [upload=463799]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:17:06.589611337 (528h48m6.460694402s) [upload=463799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:17:06.886744620 (528h48m6.757827665s) [upload=463799]: Done uploading segments... Dec 27 16:17:06.886769317 (528h48m6.757852362s) [upload=463799]: Upload finished. err: <nil> Duration: 5.839952865s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X..XXX...........X.......X..X.XX..X..X....XX.X....X..X.XXX.] duration: 6.628589538s trace: https://jaeger.dev.storj.io/trace/6d6e199525efdebc FAIL (2025-12-27 16:18:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 16:17:04.059444495 (528h48m3.93052755s) [upload=463799]: Starting upload Dec 27 16:17:04.059548012 (528h48m3.930631057s) [upload=463799]: Done starting upload Dec 27 16:17:04.059614218 (528h48m3.930697263s) [upload=463799]: Uploading segments... Dec 27 16:17:04.059805102 (528h48m3.930888147s) [upload=463799]: Got next segment. Inline: false Dec 27 16:17:04.059854916 (528h48m3.930937961s) [upload=463799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:17:04.059861759 (528h48m3.930944814s) [upload=463799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:17:04.303537741 (528h48m4.174620806s) [seg_pos={0 0}, upload=463799]: Begin upload segment... Dec 27 16:17:04.303607855 (528h48m4.17469091s) [seg_pos={0 0}, upload=463799]: Done begin upload segment. Dec 27 16:17:04.303637271 (528h48m4.174720316s) [noise=true , address=145.79.227.15:4874 , piece_id=NOL5V5JW34ZHFLUTMX2YCPW2R5ONUXLZH6IIFC3TFPSKAB56JGCQ, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.303652860 (528h48m4.174735905s) [noise=true , address=145.79.227.21:4474 , piece_id=OSOWWSVI2H4S5Y2D3SYX6GYSVYD3CWTOTJ56V4DGM7KDTH7LMODQ, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.304269284 (528h48m4.175352349s) [noise=true , address=45.91.116.7:4012 , piece_id=YQPGCR5UWAHNPU34H2LVJJXUJGRDJAJSFMH7PWUPOIIKBFUPDVCA, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.304868345 (528h48m4.17595142s) [noise=true , address=145.79.227.5:4223 , piece_id=OXYRSSWS5K354GMTIMYLF2CV4WBAAVCUBMECMCZP4OK7VTEGKN3A, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.305382454 (528h48m4.176465499s) [noise=true , address=145.79.227.8:4329 , piece_id=HNCYY223DBE3DG2VVKGEUURAONICIXDI32MVYWLI3IHBNHJDZR4Q, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.305657157 (528h48m4.176740202s) [noise=true , address=145.79.227.3:4149 , piece_id=WJCBBVQKT7LDGHTPQ7KZNIBMCOR7VHKCL3HN23WK4ISWDS4U5I7A, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.306516173 (528h48m4.177599218s) [noise=true , address=145.79.227.48:4302 , piece_id=3VZHUGCL5NSNDA3RTZDK6DNPTSBUOANENH5PEAFROZKQRTJ6ZAZQ, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.306981419 (528h48m4.178064464s) [noise=true , address=145.79.227.34:4907 , piece_id=2SWKNXYL4DSQGKDLRSCUBWWCAS7UOHPYM6OYQ3MX7FGVK5UWDZIA, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.307495368 (528h48m4.178578423s) [noise=true , address=145.79.227.45:4231 , piece_id=HEWZ3DKOG4ZJDEFUEQYHEWK5KUIRHSQC4RSVVST7PZJZS5LO2PUQ, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.307992053 (528h48m4.179075098s) [noise=true , address=145.79.227.50:4374 , piece_id=ZQZYXIDCD4FCPAVRINDZL6ADZZBMQUAGZIJA7GQTQRJJM5RFJLEA, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.308595772 (528h48m4.179678818s) [noise=true , address=145.79.227.51:4411 , piece_id=OUWBINBHJPG73DYCI56DU3Z6DJTIJRVXOLAONQFQG45J7HNMM2FA, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.309183612 (528h48m4.180266657s) [noise=true , address=145.79.227.27:4691 , piece_id=CPZC2ZSPWYVXEZO3BOIYYW5Z4REXJBZTUXCB62GK5OWLBBFDHVQQ, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.309783004 (528h48m4.180866049s) [noise=true , address=45.91.116.8:4046 , piece_id=FEHWPJYHTNY3MUGTTOCBHCPOG4QERTSX24BNQA2KE3DBUFZQ6K4Q, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.312066582 (528h48m4.183149627s) [noise=true , address=145.79.227.37:4565 , piece_id=5G5WHRDU7JJS2NBXOEGKVJVUN3ASOQ5NBPEURWY6BNGTFSDMTEHQ, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.314315424 (528h48m4.185398469s) [noise=true , address=77.90.31.18:4808 , piece_id=5CO5LPIO5W653525HP6OW3E2TPDAKUEMDPUJSM2OS7NUVXTZ5MCA, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.316040609 (528h48m4.187123644s) [noise=true , address=77.90.31.17:4773 , piece_id=64D4YBFA4FPJR6EOTLSEKQBNZIS36LY3GJHXMJ4IWJSJPY3GBFSA, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:06.023052108 (528h48m5.894135154s) [upload=463799]: Next returned nil segment Dec 27 16:17:06.023075864 (528h48m5.894158909s) [upload=463799]: Waiting for error group managing segments... Dec 27 16:17:06.299063353 (528h48m6.170146388s) [noise=true , address=145.79.227.15:4874 , piece_id=NOL5V5JW34ZHFLUTMX2YCPW2R5ONUXLZH6IIFC3TFPSKAB56JGCQ, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.306970299 (528h48m6.178053334s) [noise=true , address=145.79.227.21:4474 , piece_id=OSOWWSVI2H4S5Y2D3SYX6GYSVYD3CWTOTJ56V4DGM7KDTH7LMODQ, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.313053092 (528h48m6.184136127s) [noise=true , address=145.79.227.5:4223 , piece_id=OXYRSSWS5K354GMTIMYLF2CV4WBAAVCUBMECMCZP4OK7VTEGKN3A, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.327281685 (528h48m6.19836472s) [noise=true , address=145.79.227.8:4329 , piece_id=HNCYY223DBE3DG2VVKGEUURAONICIXDI32MVYWLI3IHBNHJDZR4Q, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.329546667 (528h48m6.200629702s) [noise=true , address=145.79.227.27:4691 , piece_id=CPZC2ZSPWYVXEZO3BOIYYW5Z4REXJBZTUXCB62GK5OWLBBFDHVQQ, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.332383259 (528h48m6.203466294s) [noise=true , address=145.79.227.3:4149 , piece_id=WJCBBVQKT7LDGHTPQ7KZNIBMCOR7VHKCL3HN23WK4ISWDS4U5I7A, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.333616738 (528h48m6.204699783s) [noise=true , address=145.79.227.45:4231 , piece_id=HEWZ3DKOG4ZJDEFUEQYHEWK5KUIRHSQC4RSVVST7PZJZS5LO2PUQ, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.337131420 (528h48m6.208214455s) [noise=true , address=145.79.227.37:4565 , piece_id=5G5WHRDU7JJS2NBXOEGKVJVUN3ASOQ5NBPEURWY6BNGTFSDMTEHQ, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.338729493 (528h48m6.209812528s) [noise=true , address=145.79.227.51:4411 , piece_id=OUWBINBHJPG73DYCI56DU3Z6DJTIJRVXOLAONQFQG45J7HNMM2FA, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.356759526 (528h48m6.227842561s) [noise=true , address=145.79.227.50:4374 , piece_id=ZQZYXIDCD4FCPAVRINDZL6ADZZBMQUAGZIJA7GQTQRJJM5RFJLEA, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.360719556 (528h48m6.231802592s) [noise=true , address=145.79.227.48:4302 , piece_id=3VZHUGCL5NSNDA3RTZDK6DNPTSBUOANENH5PEAFROZKQRTJ6ZAZQ, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.572641544 (528h48m6.443724599s) [noise=true , address=45.91.116.7:4012 , piece_id=YQPGCR5UWAHNPU34H2LVJJXUJGRDJAJSFMH7PWUPOIIKBFUPDVCA, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.588753143 (528h48m6.459836198s) [noise=true , address=45.91.116.8:4046 , piece_id=FEHWPJYHTNY3MUGTTOCBHCPOG4QERTSX24BNQA2KE3DBUFZQ6K4Q, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.588779943 (528h48m6.459862988s) [seg_pos={0 0}, upload=463799]: Segment reached optimal threshold of 13 pieces. Dec 27 16:17:06.588995193 (528h48m6.460078238s) [noise=true , address=77.90.31.18:4808 , piece_id=5CO5LPIO5W653525HP6OW3E2TPDAKUEMDPUJSM2OS7NUVXTZ5MCA, seg_pos={0 0}, upload=463799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 27 16:17:06.589002949 (528h48m6.460086004s) [noise=true , address=77.90.31.18:4808 , piece_id=5CO5LPIO5W653525HP6OW3E2TPDAKUEMDPUJSM2OS7NUVXTZ5MCA, seg_pos={0 0}, upload=463799]: Long tail cancellation due to optimal threshold Dec 27 16:17:06.589334710 (528h48m6.460417755s) [noise=true , address=145.79.227.34:4907 , piece_id=2SWKNXYL4DSQGKDLRSCUBWWCAS7UOHPYM6OYQ3MX7FGVK5UWDZIA, seg_pos={0 0}, upload=463799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 27 16:17:06.589343436 (528h48m6.460426481s) [noise=true , address=145.79.227.34:4907 , piece_id=2SWKNXYL4DSQGKDLRSCUBWWCAS7UOHPYM6OYQ3MX7FGVK5UWDZIA, seg_pos={0 0}, upload=463799]: Long tail cancellation due to optimal threshold Dec 27 16:17:06.589498211 (528h48m6.460581257s) [noise=true , address=77.90.31.17:4773 , piece_id=64D4YBFA4FPJR6EOTLSEKQBNZIS36LY3GJHXMJ4IWJSJPY3GBFSA, seg_pos={0 0}, upload=463799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:56700->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:56700->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56700->77.90.31.17:4773: use of closed network connection Dec 27 16:17:06.589507589 (528h48m6.460590644s) [noise=true , address=77.90.31.17:4773 , piece_id=64D4YBFA4FPJR6EOTLSEKQBNZIS36LY3GJHXMJ4IWJSJPY3GBFSA, seg_pos={0 0}, upload=463799]: Long tail cancellation due to optimal threshold Dec 27 16:17:06.589525594 (528h48m6.460608649s) [seg_pos={0 0}, upload=463799]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:17:06.589605305 (528h48m6.46068836s) [upload=463799]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:17:06.589611337 (528h48m6.460694402s) [upload=463799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:17:06.886744620 (528h48m6.757827665s) [upload=463799]: Done uploading segments... Dec 27 16:17:06.886769317 (528h48m6.757852362s) [upload=463799]: Upload finished. err: <nil> Duration: 6.628624855s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X................XX...................................] duration: 33.233788215s trace: https://jaeger.dev.storj.io/trace/9fd07cafe8f934a FAIL (2025-12-27 16:17:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315411726 (528h16m20.186494771s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315905206 (528h16m20.186988251s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316356304 (528h16m20.187439339s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316793537 (528h16m20.187876582s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.317233315 (528h16m20.18831636s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318042255 (528h16m20.1891253s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318150922 (528h16m20.189233967s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318513712 (528h16m20.189596747s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318955464 (528h16m20.190038509s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319037570 (528h16m20.190120605s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319389541 (528h16m20.190472576s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319874033 (528h16m20.190957078s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320378844 (528h16m20.191461879s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320898604 (528h16m20.191981649s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.321434254 (528h16m20.192517289s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:21.620856000 (528h16m21.491939035s) [upload=463344]: Next returned nil segment Dec 27 15:45:21.620875698 (528h16m21.491958743s) [upload=463344]: Waiting for error group managing segments... Dec 27 15:45:21.953178480 (528h16m21.824261516s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.194632602 (528h16m22.065715647s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.195118968 (528h16m22.066202013s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.204573761 (528h16m22.075656806s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.213413343 (528h16m22.084496388s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.372654386 (528h16m22.243737431s) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.473126101 (528h16m22.344209146s) [noise=true , address=77.90.31.12:4615 , piece_id=NTYP2JTUJYSBE4NBMQHTEAY2DA333CR7V3622PVQ3OGRMSPCJK3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.484926481 (528h16m22.356009536s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.563981276 (528h16m22.435064311s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.893470090 (528h16m22.764553126s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.105921755 (528h16m22.9770048s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.153398930 (528h16m23.024481975s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.218328398 (528h16m23.089411443s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244693782 (528h16m23.115776827s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244714231 (528h16m23.115797266s) [seg_pos={0 0}, upload=463344]: Segment reached optimal threshold of 14 pieces. Dec 27 15:45:23.244907037 (528h16m23.115990073s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.244913159 (528h16m23.115996204s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245080929 (528h16m23.116163964s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245087220 (528h16m23.116170275s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245280880 (528h16m23.116363925s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection Dec 27 15:45:23.245370850 (528h16m23.116453885s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245453588 (528h16m23.116536644s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f2MspBPykwYCWAYzJ9PCAXop6mPNarJt3fa6fWZoLipZZQdy1): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245474488 (528h16m23.116557533s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245514535 (528h16m23.11659757s) [seg_pos={0 0}, upload=463344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:45:23.246027922 (528h16m23.117110977s) [upload=463344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:45:23.246044133 (528h16m23.117127168s) [upload=463344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:45:23.560988923 (528h16m23.432071968s) [upload=463344]: Done uploading segments... Dec 27 15:45:23.561009573 (528h16m23.432092618s) [upload=463344]: Upload finished. err: <nil> Duration: 33.233801361s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [...XX.XXXX....XXXX..XX..XX......X......XX...X.............X.] duration: 5.001134458s trace: https://jaeger.dev.storj.io/trace/6b9e6d51cd89450e FAIL (2025-12-27 16:17:55 UTC) orbiter warning: context deadline exceeded Duration: 5.001160629s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..X..XXX...........X.......X..X.XX..X..X....XX.X....X..X.XXX] duration: 30.002666532s trace: https://jaeger.dev.storj.io/trace/6d6e199525efdebc FAIL (2025-12-27 16:17:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 16:17:04.059444495 (528h48m3.93052755s) [upload=463799]: Starting upload Dec 27 16:17:04.059548012 (528h48m3.930631057s) [upload=463799]: Done starting upload Dec 27 16:17:04.059614218 (528h48m3.930697263s) [upload=463799]: Uploading segments... Dec 27 16:17:04.059805102 (528h48m3.930888147s) [upload=463799]: Got next segment. Inline: false Dec 27 16:17:04.059854916 (528h48m3.930937961s) [upload=463799]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:17:04.059861759 (528h48m3.930944814s) [upload=463799]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:17:04.303537741 (528h48m4.174620806s) [seg_pos={0 0}, upload=463799]: Begin upload segment... Dec 27 16:17:04.303607855 (528h48m4.17469091s) [seg_pos={0 0}, upload=463799]: Done begin upload segment. Dec 27 16:17:04.303637271 (528h48m4.174720316s) [noise=true , address=145.79.227.15:4874 , piece_id=NOL5V5JW34ZHFLUTMX2YCPW2R5ONUXLZH6IIFC3TFPSKAB56JGCQ, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.303652860 (528h48m4.174735905s) [noise=true , address=145.79.227.21:4474 , piece_id=OSOWWSVI2H4S5Y2D3SYX6GYSVYD3CWTOTJ56V4DGM7KDTH7LMODQ, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.304269284 (528h48m4.175352349s) [noise=true , address=45.91.116.7:4012 , piece_id=YQPGCR5UWAHNPU34H2LVJJXUJGRDJAJSFMH7PWUPOIIKBFUPDVCA, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.304868345 (528h48m4.17595142s) [noise=true , address=145.79.227.5:4223 , piece_id=OXYRSSWS5K354GMTIMYLF2CV4WBAAVCUBMECMCZP4OK7VTEGKN3A, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.305382454 (528h48m4.176465499s) [noise=true , address=145.79.227.8:4329 , piece_id=HNCYY223DBE3DG2VVKGEUURAONICIXDI32MVYWLI3IHBNHJDZR4Q, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.305657157 (528h48m4.176740202s) [noise=true , address=145.79.227.3:4149 , piece_id=WJCBBVQKT7LDGHTPQ7KZNIBMCOR7VHKCL3HN23WK4ISWDS4U5I7A, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.306516173 (528h48m4.177599218s) [noise=true , address=145.79.227.48:4302 , piece_id=3VZHUGCL5NSNDA3RTZDK6DNPTSBUOANENH5PEAFROZKQRTJ6ZAZQ, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.306981419 (528h48m4.178064464s) [noise=true , address=145.79.227.34:4907 , piece_id=2SWKNXYL4DSQGKDLRSCUBWWCAS7UOHPYM6OYQ3MX7FGVK5UWDZIA, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.307495368 (528h48m4.178578423s) [noise=true , address=145.79.227.45:4231 , piece_id=HEWZ3DKOG4ZJDEFUEQYHEWK5KUIRHSQC4RSVVST7PZJZS5LO2PUQ, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.307992053 (528h48m4.179075098s) [noise=true , address=145.79.227.50:4374 , piece_id=ZQZYXIDCD4FCPAVRINDZL6ADZZBMQUAGZIJA7GQTQRJJM5RFJLEA, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.308595772 (528h48m4.179678818s) [noise=true , address=145.79.227.51:4411 , piece_id=OUWBINBHJPG73DYCI56DU3Z6DJTIJRVXOLAONQFQG45J7HNMM2FA, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.309183612 (528h48m4.180266657s) [noise=true , address=145.79.227.27:4691 , piece_id=CPZC2ZSPWYVXEZO3BOIYYW5Z4REXJBZTUXCB62GK5OWLBBFDHVQQ, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.309783004 (528h48m4.180866049s) [noise=true , address=45.91.116.8:4046 , piece_id=FEHWPJYHTNY3MUGTTOCBHCPOG4QERTSX24BNQA2KE3DBUFZQ6K4Q, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.312066582 (528h48m4.183149627s) [noise=true , address=145.79.227.37:4565 , piece_id=5G5WHRDU7JJS2NBXOEGKVJVUN3ASOQ5NBPEURWY6BNGTFSDMTEHQ, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.314315424 (528h48m4.185398469s) [noise=true , address=77.90.31.18:4808 , piece_id=5CO5LPIO5W653525HP6OW3E2TPDAKUEMDPUJSM2OS7NUVXTZ5MCA, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:04.316040609 (528h48m4.187123644s) [noise=true , address=77.90.31.17:4773 , piece_id=64D4YBFA4FPJR6EOTLSEKQBNZIS36LY3GJHXMJ4IWJSJPY3GBFSA, seg_pos={0 0}, upload=463799]: Uploading piece... Dec 27 16:17:06.023052108 (528h48m5.894135154s) [upload=463799]: Next returned nil segment Dec 27 16:17:06.023075864 (528h48m5.894158909s) [upload=463799]: Waiting for error group managing segments... Dec 27 16:17:06.299063353 (528h48m6.170146388s) [noise=true , address=145.79.227.15:4874 , piece_id=NOL5V5JW34ZHFLUTMX2YCPW2R5ONUXLZH6IIFC3TFPSKAB56JGCQ, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.306970299 (528h48m6.178053334s) [noise=true , address=145.79.227.21:4474 , piece_id=OSOWWSVI2H4S5Y2D3SYX6GYSVYD3CWTOTJ56V4DGM7KDTH7LMODQ, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.313053092 (528h48m6.184136127s) [noise=true , address=145.79.227.5:4223 , piece_id=OXYRSSWS5K354GMTIMYLF2CV4WBAAVCUBMECMCZP4OK7VTEGKN3A, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.327281685 (528h48m6.19836472s) [noise=true , address=145.79.227.8:4329 , piece_id=HNCYY223DBE3DG2VVKGEUURAONICIXDI32MVYWLI3IHBNHJDZR4Q, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.329546667 (528h48m6.200629702s) [noise=true , address=145.79.227.27:4691 , piece_id=CPZC2ZSPWYVXEZO3BOIYYW5Z4REXJBZTUXCB62GK5OWLBBFDHVQQ, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.332383259 (528h48m6.203466294s) [noise=true , address=145.79.227.3:4149 , piece_id=WJCBBVQKT7LDGHTPQ7KZNIBMCOR7VHKCL3HN23WK4ISWDS4U5I7A, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.333616738 (528h48m6.204699783s) [noise=true , address=145.79.227.45:4231 , piece_id=HEWZ3DKOG4ZJDEFUEQYHEWK5KUIRHSQC4RSVVST7PZJZS5LO2PUQ, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.337131420 (528h48m6.208214455s) [noise=true , address=145.79.227.37:4565 , piece_id=5G5WHRDU7JJS2NBXOEGKVJVUN3ASOQ5NBPEURWY6BNGTFSDMTEHQ, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.338729493 (528h48m6.209812528s) [noise=true , address=145.79.227.51:4411 , piece_id=OUWBINBHJPG73DYCI56DU3Z6DJTIJRVXOLAONQFQG45J7HNMM2FA, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.356759526 (528h48m6.227842561s) [noise=true , address=145.79.227.50:4374 , piece_id=ZQZYXIDCD4FCPAVRINDZL6ADZZBMQUAGZIJA7GQTQRJJM5RFJLEA, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.360719556 (528h48m6.231802592s) [noise=true , address=145.79.227.48:4302 , piece_id=3VZHUGCL5NSNDA3RTZDK6DNPTSBUOANENH5PEAFROZKQRTJ6ZAZQ, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.572641544 (528h48m6.443724599s) [noise=true , address=45.91.116.7:4012 , piece_id=YQPGCR5UWAHNPU34H2LVJJXUJGRDJAJSFMH7PWUPOIIKBFUPDVCA, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.588753143 (528h48m6.459836198s) [noise=true , address=45.91.116.8:4046 , piece_id=FEHWPJYHTNY3MUGTTOCBHCPOG4QERTSX24BNQA2KE3DBUFZQ6K4Q, seg_pos={0 0}, upload=463799]: Done uploading piece. err: <nil> Dec 27 16:17:06.588779943 (528h48m6.459862988s) [seg_pos={0 0}, upload=463799]: Segment reached optimal threshold of 13 pieces. Dec 27 16:17:06.588995193 (528h48m6.460078238s) [noise=true , address=77.90.31.18:4808 , piece_id=5CO5LPIO5W653525HP6OW3E2TPDAKUEMDPUJSM2OS7NUVXTZ5MCA, seg_pos={0 0}, upload=463799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 27 16:17:06.589002949 (528h48m6.460086004s) [noise=true , address=77.90.31.18:4808 , piece_id=5CO5LPIO5W653525HP6OW3E2TPDAKUEMDPUJSM2OS7NUVXTZ5MCA, seg_pos={0 0}, upload=463799]: Long tail cancellation due to optimal threshold Dec 27 16:17:06.589334710 (528h48m6.460417755s) [noise=true , address=145.79.227.34:4907 , piece_id=2SWKNXYL4DSQGKDLRSCUBWWCAS7UOHPYM6OYQ3MX7FGVK5UWDZIA, seg_pos={0 0}, upload=463799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 27 16:17:06.589343436 (528h48m6.460426481s) [noise=true , address=145.79.227.34:4907 , piece_id=2SWKNXYL4DSQGKDLRSCUBWWCAS7UOHPYM6OYQ3MX7FGVK5UWDZIA, seg_pos={0 0}, upload=463799]: Long tail cancellation due to optimal threshold Dec 27 16:17:06.589498211 (528h48m6.460581257s) [noise=true , address=77.90.31.17:4773 , piece_id=64D4YBFA4FPJR6EOTLSEKQBNZIS36LY3GJHXMJ4IWJSJPY3GBFSA, seg_pos={0 0}, upload=463799]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: write tcp 188.166.225.170:56700->77.90.31.17:4773: use of closed network connection; write tcp 188.166.225.170:56700->77.90.31.17:4773: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56700->77.90.31.17:4773: use of closed network connection Dec 27 16:17:06.589507589 (528h48m6.460590644s) [noise=true , address=77.90.31.17:4773 , piece_id=64D4YBFA4FPJR6EOTLSEKQBNZIS36LY3GJHXMJ4IWJSJPY3GBFSA, seg_pos={0 0}, upload=463799]: Long tail cancellation due to optimal threshold Dec 27 16:17:06.589525594 (528h48m6.460608649s) [seg_pos={0 0}, upload=463799]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:17:06.589605305 (528h48m6.46068836s) [upload=463799]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:17:06.589611337 (528h48m6.460694402s) [upload=463799]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:17:06.886744620 (528h48m6.757827665s) [upload=463799]: Done uploading segments... Dec 27 16:17:06.886769317 (528h48m6.757852362s) [upload=463799]: Upload finished. err: <nil> Duration: 30.002685407s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 4 out of 60 is failed (threshold=3) [X......X................XX..................................] duration: 53.776462131s trace: https://jaeger.dev.storj.io/trace/9fd07cafe8f934a FAIL (2025-12-27 16:17:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315411726 (528h16m20.186494771s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315905206 (528h16m20.186988251s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316356304 (528h16m20.187439339s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316793537 (528h16m20.187876582s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.317233315 (528h16m20.18831636s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318042255 (528h16m20.1891253s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318150922 (528h16m20.189233967s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318513712 (528h16m20.189596747s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318955464 (528h16m20.190038509s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319037570 (528h16m20.190120605s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319389541 (528h16m20.190472576s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319874033 (528h16m20.190957078s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320378844 (528h16m20.191461879s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320898604 (528h16m20.191981649s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.321434254 (528h16m20.192517289s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:21.620856000 (528h16m21.491939035s) [upload=463344]: Next returned nil segment Dec 27 15:45:21.620875698 (528h16m21.491958743s) [upload=463344]: Waiting for error group managing segments... Dec 27 15:45:21.953178480 (528h16m21.824261516s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.194632602 (528h16m22.065715647s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.195118968 (528h16m22.066202013s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.204573761 (528h16m22.075656806s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.213413343 (528h16m22.084496388s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.372654386 (528h16m22.243737431s) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.473126101 (528h16m22.344209146s) [noise=true , address=77.90.31.12:4615 , piece_id=NTYP2JTUJYSBE4NBMQHTEAY2DA333CR7V3622PVQ3OGRMSPCJK3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.484926481 (528h16m22.356009536s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.563981276 (528h16m22.435064311s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.893470090 (528h16m22.764553126s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.105921755 (528h16m22.9770048s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.153398930 (528h16m23.024481975s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.218328398 (528h16m23.089411443s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244693782 (528h16m23.115776827s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244714231 (528h16m23.115797266s) [seg_pos={0 0}, upload=463344]: Segment reached optimal threshold of 14 pieces. Dec 27 15:45:23.244907037 (528h16m23.115990073s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.244913159 (528h16m23.115996204s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245080929 (528h16m23.116163964s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245087220 (528h16m23.116170275s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245280880 (528h16m23.116363925s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection Dec 27 15:45:23.245370850 (528h16m23.116453885s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245453588 (528h16m23.116536644s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f2MspBPykwYCWAYzJ9PCAXop6mPNarJt3fa6fWZoLipZZQdy1): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245474488 (528h16m23.116557533s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245514535 (528h16m23.11659757s) [seg_pos={0 0}, upload=463344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:45:23.246027922 (528h16m23.117110977s) [upload=463344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:45:23.246044133 (528h16m23.117127168s) [upload=463344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:45:23.560988923 (528h16m23.432071968s) [upload=463344]: Done uploading segments... Dec 27 15:45:23.561009573 (528h16m23.432092618s) [upload=463344]: Upload finished. err: <nil> Duration: 53.776478382s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [X...XX.XXXX....XXXX..XX..XX......X......XX...X.............X] duration: 5.001662103s trace: https://jaeger.dev.storj.io/trace/6b9e6d51cd89450e FAIL (2025-12-27 16:17:14 UTC) orbiter warning: context deadline exceeded Duration: 5.001687911s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [X..X..XXX...........X.......X..X.XX..X..X....XX.X....X..X.XX] duration: 30.002414242s trace: https://jaeger.dev.storj.io/trace/6fa47ab34c3a8161 FAIL (2025-12-27 16:16:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 27 16:16:18.692420731 (528h47m18.563503766s) [upload=463786]: Starting upload Dec 27 16:16:18.692478620 (528h47m18.563561655s) [upload=463786]: Done starting upload Dec 27 16:16:18.692571457 (528h47m18.563654502s) [upload=463786]: Uploading segments... Dec 27 16:16:18.692703128 (528h47m18.563786173s) [upload=463786]: Got next segment. Inline: false Dec 27 16:16:18.692746791 (528h47m18.563829836s) [upload=463786]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:16:18.692753594 (528h47m18.563836639s) [upload=463786]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:16:18.938534133 (528h47m18.809617178s) [seg_pos={0 0}, upload=463786]: Begin upload segment... Dec 27 16:16:18.938586714 (528h47m18.809669759s) [seg_pos={0 0}, upload=463786]: Done begin upload segment. Dec 27 16:16:18.938619044 (528h47m18.809702079s) [noise=true , address=145.79.227.10:4402 , piece_id=H7EMZXIBKWMV6Z2LFC4URX75QXF6LIMTAUBKO3GVLLYUF4OKYYAA, seg_pos={0 0}, upload=463786]: Uploading piece... Dec 27 16:16:18.938764823 (528h47m18.809847858s) [noise=true , address=145.79.227.23:4545 , piece_id=MAOLRSIRKAXMFPQRMBVNXURBXBXVK6POHY7MR7E4UNVAMVWB2SXA, seg_pos={0 0}, upload=463786]: Uploading piece... Dec 27 16:16:18.939314088 (528h47m18.810397133s) [noise=true , address=145.79.227.4:4186 , piece_id=6FGMJ5W7SYBKE63INETA4GNMYSQ5I65RBTR7DWMGUK7Q5AHBWGIA, seg_pos={0 0}, upload=463786]: Uploading piece... Dec 27 16:16:18.939698790 (528h47m18.810781835s) [noise=true , address=145.79.227.37:4566 , piece_id=XJQJCAGZTETCEEFO2O2BSUB6EKPFV5UF72EVFV5ATLU7N25VH2NQ, seg_pos={0 0}, upload=463786]: Uploading piece... Dec 27 16:16:18.940074065 (528h47m18.81115711s) [noise=true , address=145.79.227.13:4802 , piece_id=SVRJT2SW3TF4EJFJDI5RLMWMN2WXMTCCMOO6B74MYAJCH4OXUPRA, seg_pos={0 0}, upload=463786]: Uploading piece... Dec 27 16:16:18.940437197 (528h47m18.811520242s) [noise=true , address=45.91.116.12:4228 , piece_id=MO3YSM5ELW5L3DSKATVM3PZJPQ3QA25NF3V42VJPCHLTLD5HYBJA, seg_pos={0 0}, upload=463786]: Uploading piece... Dec 27 16:16:18.940788396 (528h47m18.811871441s) [noise=true , address=77.90.31.15:4697 , piece_id=AQSTJYXZWFDF4TJOUAAKN3SWQZEES52UCBULG2T62Y5CMX5KG5UQ, seg_pos={0 0}, upload=463786]: Uploading piece... Dec 27 16:16:18.941152038 (528h47m18.812235083s) [noise=true , address=145.79.227.47:4265 , piece_id=4G4MFVIZTP4VAOZSDKUHZM6ORNVIQS3BCM5UU6OWULOMDXSMNB3Q, seg_pos={0 0}, upload=463786]: Uploading piece... Dec 27 16:16:18.941595232 (528h47m18.812678277s) [noise=true , address=145.79.227.36:4517 , piece_id=RS663MU4GJ5RW2GQI6TKRANZ6ODEMK33O47U5DWX4LFFEVLC66KA, seg_pos={0 0}, upload=463786]: Uploading piece... Dec 27 16:16:18.942934822 (528h47m18.814017857s) [noise=true , address=45.91.116.6:4974 , piece_id=W7EXNB4WOF6QCYG6ZRNW75RW5KTRHUJDJ6HXCDGG6AJBLLBNTCXQ, seg_pos={0 0}, upload=463786]: Uploading piece... Dec 27 16:16:18.944302838 (528h47m18.815385883s) [noise=true , address=45.91.116.14:4191 , piece_id=SYR3VQ53FF3DKEYKQZVMGJWTWO5UMLWCU4IOIOU33Y6WI5ETV4RA, seg_pos={0 0}, upload=463786]: Uploading piece... Dec 27 16:16:18.946461458 (528h47m18.817544503s) [noise=true , address=45.91.116.7:4010 , piece_id=WUTKAPMR6XHFMAJ3QG7ZUYTREUG3VMAMUGV7QB5H2DYQNRXNZLOA, seg_pos={0 0}, upload=463786]: Uploading piece... Dec 27 16:16:18.947839742 (528h47m18.818922777s) [noise=true , address=45.91.116.9:4262 , piece_id=W477FUGTCC2JFCW3M5EU76VML265G4O2UIU3GJGNHUXHY33T6SJA, seg_pos={0 0}, upload=463786]: Uploading piece... Dec 27 16:16:18.949192488 (528h47m18.820275533s) [noise=true , address=145.79.227.5:4223 , piece_id=EUMEAGQAB3FK7X5UCMRS4MOEGLSIKEG24QDGDAOXXDGD6MBXJBVQ, seg_pos={0 0}, upload=463786]: Uploading piece... Dec 27 16:16:18.950491743 (528h47m18.821574788s) [noise=true , address=145.79.227.38:4978 , piece_id=2Y7PVRLV2ISG5UJUU4JRYNB2IDERMFKC4WZ2KKIV7WKCESIIA3TQ, seg_pos={0 0}, upload=463786]: Uploading piece... Dec 27 16:16:18.951737514 (528h47m18.822820559s) [noise=true , address=145.79.227.45:4229 , piece_id=JOUFGTFVJMFWFWF27D6RMZ5FFPGI3HBWNCBBKXGCS5BKI7HVFMBA, seg_pos={0 0}, upload=463786]: Uploading piece... Dec 27 16:16:20.655760072 (528h47m20.526843127s) [upload=463786]: Next returned nil segment Dec 27 16:16:20.655780421 (528h47m20.526863467s) [upload=463786]: Waiting for error group managing segments... Dec 27 16:16:20.943729187 (528h47m20.814812422s) [noise=true , address=145.79.227.37:4566 , piece_id=XJQJCAGZTETCEEFO2O2BSUB6EKPFV5UF72EVFV5ATLU7N25VH2NQ, seg_pos={0 0}, upload=463786]: Done uploading piece. err: <nil> Dec 27 16:16:20.949094593 (528h47m20.820177638s) [noise=true , address=145.79.227.36:4517 , piece_id=RS663MU4GJ5RW2GQI6TKRANZ6ODEMK33O47U5DWX4LFFEVLC66KA, seg_pos={0 0}, upload=463786]: Done uploading piece. err: <nil> Dec 27 16:16:20.949506087 (528h47m20.820589122s) [noise=true , address=145.79.227.13:4802 , piece_id=SVRJT2SW3TF4EJFJDI5RLMWMN2WXMTCCMOO6B74MYAJCH4OXUPRA, seg_pos={0 0}, upload=463786]: Done uploading piece. err: <nil> Dec 27 16:16:20.952813114 (528h47m20.823896149s) [noise=true , address=145.79.227.10:4402 , piece_id=H7EMZXIBKWMV6Z2LFC4URX75QXF6LIMTAUBKO3GVLLYUF4OKYYAA, seg_pos={0 0}, upload=463786]: Done uploading piece. err: <nil> Dec 27 16:16:20.954268506 (528h47m20.825351541s) [noise=true , address=145.79.227.23:4545 , piece_id=MAOLRSIRKAXMFPQRMBVNXURBXBXVK6POHY7MR7E4UNVAMVWB2SXA, seg_pos={0 0}, upload=463786]: Done uploading piece. err: <nil> Dec 27 16:16:20.958969026 (528h47m20.830052071s) [noise=true , address=145.79.227.38:4978 , piece_id=2Y7PVRLV2ISG5UJUU4JRYNB2IDERMFKC4WZ2KKIV7WKCESIIA3TQ, seg_pos={0 0}, upload=463786]: Done uploading piece. err: <nil> Dec 27 16:16:20.959558689 (528h47m20.830641734s) [noise=true , address=145.79.227.5:4223 , piece_id=EUMEAGQAB3FK7X5UCMRS4MOEGLSIKEG24QDGDAOXXDGD6MBXJBVQ, seg_pos={0 0}, upload=463786]: Done uploading piece. err: <nil> Dec 27 16:16:20.961397561 (528h47m20.832480606s) [noise=true , address=145.79.227.4:4186 , piece_id=6FGMJ5W7SYBKE63INETA4GNMYSQ5I65RBTR7DWMGUK7Q5AHBWGIA, seg_pos={0 0}, upload=463786]: Done uploading piece. err: <nil> Dec 27 16:16:20.987096756 (528h47m20.858179801s) [noise=true , address=145.79.227.45:4229 , piece_id=JOUFGTFVJMFWFWF27D6RMZ5FFPGI3HBWNCBBKXGCS5BKI7HVFMBA, seg_pos={0 0}, upload=463786]: Done uploading piece. err: <nil> Dec 27 16:16:20.988006448 (528h47m20.859089493s) [noise=true , address=145.79.227.47:4265 , piece_id=4G4MFVIZTP4VAOZSDKUHZM6ORNVIQS3BCM5UU6OWULOMDXSMNB3Q, seg_pos={0 0}, upload=463786]: Done uploading piece. err: <nil> Dec 27 16:16:21.168476024 (528h47m21.039559069s) [noise=true , address=45.91.116.9:4262 , piece_id=W477FUGTCC2JFCW3M5EU76VML265G4O2UIU3GJGNHUXHY33T6SJA, seg_pos={0 0}, upload=463786]: Done uploading piece. err: <nil> Dec 27 16:16:21.229260078 (528h47m21.100343123s) [noise=true , address=77.90.31.15:4697 , piece_id=AQSTJYXZWFDF4TJOUAAKN3SWQZEES52UCBULG2T62Y5CMX5KG5UQ, seg_pos={0 0}, upload=463786]: Done uploading piece. err: <nil> Dec 27 16:16:21.291832125 (528h47m21.16291518s) [noise=true , address=45.91.116.6:4974 , piece_id=W7EXNB4WOF6QCYG6ZRNW75RW5KTRHUJDJ6HXCDGG6AJBLLBNTCXQ, seg_pos={0 0}, upload=463786]: Done uploading piece. err: <nil> Dec 27 16:16:21.291851953 (528h47m21.162934988s) [seg_pos={0 0}, upload=463786]: Segment reached optimal threshold of 13 pieces. Dec 27 16:16:21.292081210 (528h47m21.163164265s) [noise=true , address=45.91.116.7:4010 , piece_id=WUTKAPMR6XHFMAJ3QG7ZUYTREUG3VMAMUGV7QB5H2DYQNRXNZLOA, seg_pos={0 0}, upload=463786]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 16:16:21.292091279 (528h47m21.163174314s) [noise=true , address=45.91.116.7:4010 , piece_id=WUTKAPMR6XHFMAJ3QG7ZUYTREUG3VMAMUGV7QB5H2DYQNRXNZLOA, seg_pos={0 0}, upload=463786]: Long tail cancellation due to optimal threshold Dec 27 16:16:21.292127758 (528h47m21.163210814s) [noise=true , address=45.91.116.14:4191 , piece_id=SYR3VQ53FF3DKEYKQZVMGJWTWO5UMLWCU4IOIOU33Y6WI5ETV4RA, seg_pos={0 0}, upload=463786]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 27 16:16:21.292136575 (528h47m21.16321961s) [noise=true , address=45.91.116.14:4191 , piece_id=SYR3VQ53FF3DKEYKQZVMGJWTWO5UMLWCU4IOIOU33Y6WI5ETV4RA, seg_pos={0 0}, upload=463786]: Long tail cancellation due to optimal threshold Dec 27 16:16:21.292283625 (528h47m21.16336667s) [noise=true , address=45.91.116.12:4228 , piece_id=MO3YSM5ELW5L3DSKATVM3PZJPQ3QA25NF3V42VJPCHLTLD5HYBJA, seg_pos={0 0}, upload=463786]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:60496->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:60496->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60496->45.91.116.12:4228: use of closed network connection Dec 27 16:16:21.292290759 (528h47m21.163373804s) [noise=true , address=45.91.116.12:4228 , piece_id=MO3YSM5ELW5L3DSKATVM3PZJPQ3QA25NF3V42VJPCHLTLD5HYBJA, seg_pos={0 0}, upload=463786]: Long tail cancellation due to optimal threshold Dec 27 16:16:21.292310867 (528h47m21.163393912s) [seg_pos={0 0}, upload=463786]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:16:21.292390839 (528h47m21.163473904s) [upload=463786]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:16:21.292398194 (528h47m21.163481239s) [upload=463786]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:16:21.580831953 (528h47m21.451914998s) [upload=463786]: Done uploading segments... Dec 27 16:16:21.580854475 (528h47m21.45193751s) [upload=463786]: Upload finished. err: <nil> Duration: 30.002437336s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [XX......X................XX.................................] duration: 55.055887641s trace: https://jaeger.dev.storj.io/trace/9fd07cafe8f934a FAIL (2025-12-27 16:16:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315411726 (528h16m20.186494771s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315905206 (528h16m20.186988251s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316356304 (528h16m20.187439339s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316793537 (528h16m20.187876582s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.317233315 (528h16m20.18831636s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318042255 (528h16m20.1891253s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318150922 (528h16m20.189233967s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318513712 (528h16m20.189596747s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318955464 (528h16m20.190038509s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319037570 (528h16m20.190120605s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319389541 (528h16m20.190472576s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319874033 (528h16m20.190957078s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320378844 (528h16m20.191461879s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320898604 (528h16m20.191981649s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.321434254 (528h16m20.192517289s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:21.620856000 (528h16m21.491939035s) [upload=463344]: Next returned nil segment Dec 27 15:45:21.620875698 (528h16m21.491958743s) [upload=463344]: Waiting for error group managing segments... Dec 27 15:45:21.953178480 (528h16m21.824261516s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.194632602 (528h16m22.065715647s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.195118968 (528h16m22.066202013s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.204573761 (528h16m22.075656806s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.213413343 (528h16m22.084496388s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.372654386 (528h16m22.243737431s) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.473126101 (528h16m22.344209146s) [noise=true , address=77.90.31.12:4615 , piece_id=NTYP2JTUJYSBE4NBMQHTEAY2DA333CR7V3622PVQ3OGRMSPCJK3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.484926481 (528h16m22.356009536s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.563981276 (528h16m22.435064311s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.893470090 (528h16m22.764553126s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.105921755 (528h16m22.9770048s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.153398930 (528h16m23.024481975s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.218328398 (528h16m23.089411443s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244693782 (528h16m23.115776827s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244714231 (528h16m23.115797266s) [seg_pos={0 0}, upload=463344]: Segment reached optimal threshold of 14 pieces. Dec 27 15:45:23.244907037 (528h16m23.115990073s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.244913159 (528h16m23.115996204s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245080929 (528h16m23.116163964s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245087220 (528h16m23.116170275s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245280880 (528h16m23.116363925s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection Dec 27 15:45:23.245370850 (528h16m23.116453885s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245453588 (528h16m23.116536644s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f2MspBPykwYCWAYzJ9PCAXop6mPNarJt3fa6fWZoLipZZQdy1): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245474488 (528h16m23.116557533s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245514535 (528h16m23.11659757s) [seg_pos={0 0}, upload=463344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:45:23.246027922 (528h16m23.117110977s) [upload=463344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:45:23.246044133 (528h16m23.117127168s) [upload=463344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:45:23.560988923 (528h16m23.432071968s) [upload=463344]: Done uploading segments... Dec 27 15:45:23.561009573 (528h16m23.432092618s) [upload=463344]: Upload finished. err: <nil> Duration: 55.055901609s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [.X...XX.XXXX....XXXX..XX..XX......X......XX...X.............] duration: 5.000958513s trace: https://jaeger.dev.storj.io/trace/36696332d68ad959 FAIL (2025-12-27 16:16:14 UTC) orbiter warning: context deadline exceeded Duration: 5.001024007s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X..X..XXX...........X.......X..X.XX..X..X....XX.X....X..X.X] duration: 30.003340365s trace: https://jaeger.dev.storj.io/trace/6776478765e35d2a FAIL (2025-12-27 16:16:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: TYIL3HZE3AWVLHHDOMHC6NZ5UVEQQ627YG5LXQCBTUXSY74C32PQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: MJGHXVLR57DFNG74IBLVXRKKOD477O6LI4TOAD4TLQOMPJLORIVQ) context deadline exceeded ======LOGS====== Dec 27 16:15:32.680629575 (528h46m32.55171261s) [upload=463774]: Starting upload Dec 27 16:15:32.680688978 (528h46m32.551772023s) [upload=463774]: Done starting upload Dec 27 16:15:32.680784690 (528h46m32.551867735s) [upload=463774]: Uploading segments... Dec 27 16:15:32.680878078 (528h46m32.551961123s) [upload=463774]: Got next segment. Inline: false Dec 27 16:15:32.680914528 (528h46m32.551997573s) [upload=463774]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:15:32.680919868 (528h46m32.552002903s) [upload=463774]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:15:32.924435614 (528h46m32.795518659s) [seg_pos={0 0}, upload=463774]: Begin upload segment... Dec 27 16:15:32.924497272 (528h46m32.795580307s) [seg_pos={0 0}, upload=463774]: Done begin upload segment. Dec 27 16:15:32.924566744 (528h46m32.795649789s) [noise=true , address=145.79.227.14:4839 , piece_id=BFY3IY34NCQ4SN3MYSUGL7ES6LLRNCUZVXXC7YJJBG7I3ER4YUIA, seg_pos={0 0}, upload=463774]: Uploading piece... Dec 27 16:15:32.924577955 (528h46m32.795661s) [noise=true , address=45.91.116.5:4300 , piece_id=TYIL3HZE3AWVLHHDOMHC6NZ5UVEQQ627YG5LXQCBTUXSY74C32PQ, seg_pos={0 0}, upload=463774]: Uploading piece... Dec 27 16:15:32.925148972 (528h46m32.796232017s) [noise=true , address=145.79.227.16:4912 , piece_id=6BTEQT4RDYDOVUCD5SLN5LCBYOWTW4LHSXDYUWKBJQVXJ4DGGQWQ, seg_pos={0 0}, upload=463774]: Uploading piece... Dec 27 16:15:32.925157830 (528h46m32.796240875s) [noise=true , address=145.79.227.13:4802 , piece_id=AKBQIND45WSJD75P3RSP4N726XAMOAH6CBPMPKPSD7M56CJLRM5Q, seg_pos={0 0}, upload=463774]: Uploading piece... Dec 27 16:15:32.925605372 (528h46m32.796688407s) [noise=true , address=145.79.227.4:4187 , piece_id=QQQMB5YGE7FTJDCFAXALHWZ6G5KUGSR7WVB2AWVUF2NJ7S6H2W2A, seg_pos={0 0}, upload=463774]: Uploading piece... Dec 27 16:15:32.926205174 (528h46m32.797288219s) [noise=true , address=145.79.227.7:4295 , piece_id=DW3DGFW7KZG5755O3R3OCHRMWOS3SJCF5X37GM752UTHMD54B47A, seg_pos={0 0}, upload=463774]: Uploading piece... Dec 27 16:15:32.926697191 (528h46m32.797780226s) [noise=true , address=145.79.227.29:4941 , piece_id=CQD5P2SOM7PN7A74R3KNFYQZWG63WE2ISEPXB3H6USYUOZRUY3FA, seg_pos={0 0}, upload=463774]: Uploading piece... Dec 27 16:15:32.927169870 (528h46m32.798252905s) [noise=true , address=145.79.227.20:4438 , piece_id=W3LLRU2GVCJRHA7MOFPX6MXFKTEBQ44RDKBDGINU3WPO35OB3SFQ, seg_pos={0 0}, upload=463774]: Uploading piece... Dec 27 16:15:32.927735658 (528h46m32.798818703s) [noise=true , address=145.79.227.46:4603 , piece_id=4S3ERCSG3MOJWFEZXI7V5YEJUO5TVTZM7GU7S4QCQRRJTF6IPU6Q, seg_pos={0 0}, upload=463774]: Uploading piece... Dec 27 16:15:32.928224970 (528h46m32.799308015s) [noise=true , address=145.79.227.41:4085 , piece_id=XJPFY6N6SMHL5QSEK5CGG5WUDAGPRVLR6UEYXVE4OXM5RRFAYGCA, seg_pos={0 0}, upload=463774]: Uploading piece... Dec 27 16:15:32.928751122 (528h46m32.799834157s) [noise=true , address=145.79.227.11:4732 , piece_id=XSCA6RY67HJPYRN4LXHIKTD2ZOI43P4VPUB4ENOXJEYDURBHMRRQ, seg_pos={0 0}, upload=463774]: Uploading piece... Dec 27 16:15:32.930539126 (528h46m32.801622171s) [noise=true , address=145.79.227.42:4122 , piece_id=WCZUCE3OCSICCMRQPRMFVMNMMQZAOMJZH2NJZN6PIPE2NVSIKENQ, seg_pos={0 0}, upload=463774]: Uploading piece... Dec 27 16:15:32.932618927 (528h46m32.803701972s) [noise=true , address=145.79.227.40:4050 , piece_id=F7EBCEPGEY2DZ4C4GJYKB4RTIY64LUI5NU7MT3LYON6HG4LFAMXQ, seg_pos={0 0}, upload=463774]: Uploading piece... Dec 27 16:15:32.934610579 (528h46m32.805693614s) [noise=true , address=145.79.227.45:4231 , piece_id=7IPXJWSIGODATTC2AG6DA4QWL6WHZFXBKDTVJ67WJWDGXJVLYQAQ, seg_pos={0 0}, upload=463774]: Uploading piece... Dec 27 16:15:32.936460782 (528h46m32.807543817s) [noise=true , address=145.79.227.33:4869 , piece_id=NXGTA6QH6YVXW2WHUKLS7WOITPGJYMKQ72J527N2LRFUYVGCUXXA, seg_pos={0 0}, upload=463774]: Uploading piece... Dec 27 16:15:32.939459393 (528h46m32.810542478s) [noise=true , address=77.90.31.10:4519 , piece_id=MJGHXVLR57DFNG74IBLVXRKKOD477O6LI4TOAD4TLQOMPJLORIVQ, seg_pos={0 0}, upload=463774]: Uploading piece... Dec 27 16:15:34.636129910 (528h46m34.507212945s) [upload=463774]: Next returned nil segment Dec 27 16:15:34.636151181 (528h46m34.507234216s) [upload=463774]: Waiting for error group managing segments... Dec 27 16:15:34.934710480 (528h46m34.805793515s) [noise=true , address=145.79.227.13:4802 , piece_id=AKBQIND45WSJD75P3RSP4N726XAMOAH6CBPMPKPSD7M56CJLRM5Q, seg_pos={0 0}, upload=463774]: Done uploading piece. err: <nil> Dec 27 16:15:34.943202921 (528h46m34.814285966s) [noise=true , address=145.79.227.4:4187 , piece_id=QQQMB5YGE7FTJDCFAXALHWZ6G5KUGSR7WVB2AWVUF2NJ7S6H2W2A, seg_pos={0 0}, upload=463774]: Done uploading piece. err: <nil> Dec 27 16:15:34.943602432 (528h46m34.814685467s) [noise=true , address=145.79.227.45:4231 , piece_id=7IPXJWSIGODATTC2AG6DA4QWL6WHZFXBKDTVJ67WJWDGXJVLYQAQ, seg_pos={0 0}, upload=463774]: Done uploading piece. err: <nil> Dec 27 16:15:34.947904043 (528h46m34.818987078s) [noise=true , address=145.79.227.42:4122 , piece_id=WCZUCE3OCSICCMRQPRMFVMNMMQZAOMJZH2NJZN6PIPE2NVSIKENQ, seg_pos={0 0}, upload=463774]: Done uploading piece. err: <nil> Dec 27 16:15:34.957739121 (528h46m34.828822166s) [noise=true , address=145.79.227.7:4295 , piece_id=DW3DGFW7KZG5755O3R3OCHRMWOS3SJCF5X37GM752UTHMD54B47A, seg_pos={0 0}, upload=463774]: Done uploading piece. err: <nil> Dec 27 16:15:34.961652603 (528h46m34.832735638s) [noise=true , address=145.79.227.41:4085 , piece_id=XJPFY6N6SMHL5QSEK5CGG5WUDAGPRVLR6UEYXVE4OXM5RRFAYGCA, seg_pos={0 0}, upload=463774]: Done uploading piece. err: <nil> Dec 27 16:15:34.962197150 (528h46m34.833280195s) [noise=true , address=145.79.227.16:4912 , piece_id=6BTEQT4RDYDOVUCD5SLN5LCBYOWTW4LHSXDYUWKBJQVXJ4DGGQWQ, seg_pos={0 0}, upload=463774]: Done uploading piece. err: <nil> Dec 27 16:15:34.962623622 (528h46m34.833706657s) [noise=true , address=145.79.227.20:4438 , piece_id=W3LLRU2GVCJRHA7MOFPX6MXFKTEBQ44RDKBDGINU3WPO35OB3SFQ, seg_pos={0 0}, upload=463774]: Done uploading piece. err: <nil> Dec 27 16:15:34.965342208 (528h46m34.836425243s) [noise=true , address=145.79.227.11:4732 , piece_id=XSCA6RY67HJPYRN4LXHIKTD2ZOI43P4VPUB4ENOXJEYDURBHMRRQ, seg_pos={0 0}, upload=463774]: Done uploading piece. err: <nil> Dec 27 16:15:34.969365189 (528h46m34.840448224s) [noise=true , address=145.79.227.46:4603 , piece_id=4S3ERCSG3MOJWFEZXI7V5YEJUO5TVTZM7GU7S4QCQRRJTF6IPU6Q, seg_pos={0 0}, upload=463774]: Done uploading piece. err: <nil> Dec 27 16:15:34.983059745 (528h46m34.854142781s) [noise=true , address=145.79.227.14:4839 , piece_id=BFY3IY34NCQ4SN3MYSUGL7ES6LLRNCUZVXXC7YJJBG7I3ER4YUIA, seg_pos={0 0}, upload=463774]: Done uploading piece. err: <nil> Dec 27 16:15:35.253364043 (528h46m35.124447088s) [noise=true , address=77.90.31.10:4519 , piece_id=MJGHXVLR57DFNG74IBLVXRKKOD477O6LI4TOAD4TLQOMPJLORIVQ, seg_pos={0 0}, upload=463774]: Done uploading piece. err: <nil> Dec 27 16:15:35.285142494 (528h46m35.156225539s) [noise=true , address=45.91.116.5:4300 , piece_id=TYIL3HZE3AWVLHHDOMHC6NZ5UVEQQ627YG5LXQCBTUXSY74C32PQ, seg_pos={0 0}, upload=463774]: Done uploading piece. err: <nil> Dec 27 16:15:35.285226113 (528h46m35.156309158s) [seg_pos={0 0}, upload=463774]: Segment reached optimal threshold of 13 pieces. Dec 27 16:15:35.285662334 (528h46m35.156745399s) [noise=true , address=145.79.227.40:4050 , piece_id=F7EBCEPGEY2DZ4C4GJYKB4RTIY64LUI5NU7MT3LYON6HG4LFAMXQ, seg_pos={0 0}, upload=463774]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 27 16:15:35.285672884 (528h46m35.156755929s) [noise=true , address=145.79.227.40:4050 , piece_id=F7EBCEPGEY2DZ4C4GJYKB4RTIY64LUI5NU7MT3LYON6HG4LFAMXQ, seg_pos={0 0}, upload=463774]: Long tail cancellation due to optimal threshold Dec 27 16:15:35.286130415 (528h46m35.15721347s) [noise=true , address=145.79.227.29:4941 , piece_id=CQD5P2SOM7PN7A74R3KNFYQZWG63WE2ISEPXB3H6USYUOZRUY3FA, seg_pos={0 0}, upload=463774]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF Dec 27 16:15:35.286140404 (528h46m35.157223449s) [noise=true , address=145.79.227.29:4941 , piece_id=CQD5P2SOM7PN7A74R3KNFYQZWG63WE2ISEPXB3H6USYUOZRUY3FA, seg_pos={0 0}, upload=463774]: Long tail cancellation due to optimal threshold Dec 27 16:15:35.286262076 (528h46m35.157345131s) [noise=true , address=145.79.227.33:4869 , piece_id=NXGTA6QH6YVXW2WHUKLS7WOITPGJYMKQ72J527N2LRFUYVGCUXXA, seg_pos={0 0}, upload=463774]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 27 16:15:35.286282054 (528h46m35.15736509s) [noise=true , address=145.79.227.33:4869 , piece_id=NXGTA6QH6YVXW2WHUKLS7WOITPGJYMKQ72J527N2LRFUYVGCUXXA, seg_pos={0 0}, upload=463774]: Long tail cancellation due to optimal threshold Dec 27 16:15:35.286310448 (528h46m35.157393493s) [seg_pos={0 0}, upload=463774]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:15:35.286423644 (528h46m35.157506679s) [upload=463774]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:15:35.286430657 (528h46m35.157513702s) [upload=463774]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:15:35.578889169 (528h46m35.449972204s) [upload=463774]: Done uploading segments... Dec 27 16:15:35.578908937 (528h46m35.449991982s) [upload=463774]: Upload finished. err: <nil> Duration: 30.003364644s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.XX......X................XX................................] duration: 50.967421895s trace: https://jaeger.dev.storj.io/trace/9fd07cafe8f934a FAIL (2025-12-27 16:15:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315411726 (528h16m20.186494771s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315905206 (528h16m20.186988251s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316356304 (528h16m20.187439339s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316793537 (528h16m20.187876582s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.317233315 (528h16m20.18831636s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318042255 (528h16m20.1891253s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318150922 (528h16m20.189233967s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318513712 (528h16m20.189596747s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318955464 (528h16m20.190038509s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319037570 (528h16m20.190120605s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319389541 (528h16m20.190472576s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319874033 (528h16m20.190957078s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320378844 (528h16m20.191461879s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320898604 (528h16m20.191981649s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.321434254 (528h16m20.192517289s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:21.620856000 (528h16m21.491939035s) [upload=463344]: Next returned nil segment Dec 27 15:45:21.620875698 (528h16m21.491958743s) [upload=463344]: Waiting for error group managing segments... Dec 27 15:45:21.953178480 (528h16m21.824261516s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.194632602 (528h16m22.065715647s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.195118968 (528h16m22.066202013s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.204573761 (528h16m22.075656806s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.213413343 (528h16m22.084496388s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.372654386 (528h16m22.243737431s) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.473126101 (528h16m22.344209146s) [noise=true , address=77.90.31.12:4615 , piece_id=NTYP2JTUJYSBE4NBMQHTEAY2DA333CR7V3622PVQ3OGRMSPCJK3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.484926481 (528h16m22.356009536s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.563981276 (528h16m22.435064311s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.893470090 (528h16m22.764553126s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.105921755 (528h16m22.9770048s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.153398930 (528h16m23.024481975s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.218328398 (528h16m23.089411443s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244693782 (528h16m23.115776827s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244714231 (528h16m23.115797266s) [seg_pos={0 0}, upload=463344]: Segment reached optimal threshold of 14 pieces. Dec 27 15:45:23.244907037 (528h16m23.115990073s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.244913159 (528h16m23.115996204s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245080929 (528h16m23.116163964s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245087220 (528h16m23.116170275s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245280880 (528h16m23.116363925s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection Dec 27 15:45:23.245370850 (528h16m23.116453885s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245453588 (528h16m23.116536644s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f2MspBPykwYCWAYzJ9PCAXop6mPNarJt3fa6fWZoLipZZQdy1): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245474488 (528h16m23.116557533s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245514535 (528h16m23.11659757s) [seg_pos={0 0}, upload=463344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:45:23.246027922 (528h16m23.117110977s) [upload=463344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:45:23.246044133 (528h16m23.117127168s) [upload=463344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:45:23.560988923 (528h16m23.432071968s) [upload=463344]: Done uploading segments... Dec 27 15:45:23.561009573 (528h16m23.432092618s) [upload=463344]: Upload finished. err: <nil> Duration: 50.96743999s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [..X...XX.XXXX....XXXX..XX..XX......X......XX...X............] duration: 4.978708865s trace: https://jaeger.dev.storj.io/trace/36696332d68ad959 FAIL (2025-12-27 16:15:14 UTC) orbiter warning: context deadline exceeded Duration: 4.978738271s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X.X..X..XXX...........X.......X..X.XX..X..X....XX.X....X..X.] duration: 6.350802143s trace: https://jaeger.dev.storj.io/trace/5f1261b77569ffc1 FAIL (2025-12-27 16:14:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: WJEEGB47JBD5XK7GGSXZOS3YMFGJLLYUT64LVYCYBYW273XE7MAQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: BRM46MJ7BD4GU2QYFMAZJOZSU6OHYR4KNZOYPEM277ROQ4WRQTWA) context deadline exceeded ======LOGS====== upload Dec 27 16:14:00.401050972 (528h45m0.272134028s) [upload=463754]: Uploading segments... Dec 27 16:14:00.401161513 (528h45m0.272244558s) [upload=463754]: Got next segment. Inline: false Dec 27 16:14:00.401215916 (528h45m0.272298971s) [upload=463754]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:14:00.401223741 (528h45m0.272306807s) [upload=463754]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:14:00.648483336 (528h45m0.519566372s) [seg_pos={0 0}, upload=463754]: Begin upload segment... Dec 27 16:14:00.648541638 (528h45m0.519624673s) [seg_pos={0 0}, upload=463754]: Done begin upload segment. Dec 27 16:14:00.648595961 (528h45m0.519678996s) [noise=true , address=145.79.227.4:4186 , piece_id=M3AZMAR6KQPV7S2MXAQUV7SNVTZAD3MP2F5JF55JMV6OGS6T6U5Q, seg_pos={0 0}, upload=463754]: Uploading piece... Dec 27 16:14:00.648623573 (528h45m0.519706619s) [noise=true , address=145.79.227.24:4581 , piece_id=NEUMSOWQ6SBUG3KVCQ4AQUXCV62JVAWCX4RICUIU2C5KVPTDENXQ, seg_pos={0 0}, upload=463754]: Uploading piece... Dec 27 16:14:00.649330780 (528h45m0.520413815s) [noise=true , address=145.79.227.10:4403 , piece_id=JECL6CBUJ5WEUXICHCLQ5SVQB6C276QD4ISOLZVNSKV3QF7RKP3Q, seg_pos={0 0}, upload=463754]: Uploading piece... Dec 27 16:14:00.649490454 (528h45m0.520573499s) [noise=true , address=145.79.227.37:4565 , piece_id=FQYIE47LGUVVODV7IXWAJDNIKFYGUP5AISJ2J7FRTMLS4LHX2P7A, seg_pos={0 0}, upload=463754]: Uploading piece... Dec 27 16:14:00.649784244 (528h45m0.520867289s) [noise=true , address=77.90.31.11:4555 , piece_id=WJEEGB47JBD5XK7GGSXZOS3YMFGJLLYUT64LVYCYBYW273XE7MAQ, seg_pos={0 0}, upload=463754]: Uploading piece... Dec 27 16:14:00.650253597 (528h45m0.521336633s) [noise=true , address=145.79.227.26:4655 , piece_id=MM4DZIMGAUECSLW37CCBS5ID7OO2637RJSX6DYKQSD6RDL6GZH7Q, seg_pos={0 0}, upload=463754]: Uploading piece... Dec 27 16:14:00.650365070 (528h45m0.521448105s) [noise=true , address=145.79.227.51:4411 , piece_id=D5EJV7BL6KUDHSDE3MZJ6TG5LT4O5SOB62QO52YAA7VMH4HBLAKQ, seg_pos={0 0}, upload=463754]: Uploading piece... Dec 27 16:14:00.650814495 (528h45m0.52189754s) [noise=true , address=145.79.227.27:4691 , piece_id=RALEIBLDQF7DDVJ63ROTSXYWIHJIEJHWGLCGI2S3Y5Q4YC7UYTDQ, seg_pos={0 0}, upload=463754]: Uploading piece... Dec 27 16:14:00.650934795 (528h45m0.52201783s) [noise=true , address=45.91.116.4:4937 , piece_id=QX6L4SHXTY4ZUKWE6BNHQBBMBJCS5M4TZF5RW6L2POF35I2YNXJA, seg_pos={0 0}, upload=463754]: Uploading piece... Dec 27 16:14:00.651282788 (528h45m0.522365823s) [noise=true , address=45.91.116.13:4153 , piece_id=BRM46MJ7BD4GU2QYFMAZJOZSU6OHYR4KNZOYPEM277ROQ4WRQTWA, seg_pos={0 0}, upload=463754]: Uploading piece... Dec 27 16:14:00.651569283 (528h45m0.522652318s) [noise=true , address=45.91.116.6:4975 , piece_id=BIMPZTYTSQBHEWNMKQXI5LGD2K2DIJ2S64KW7WAD62YJZPTRQWOQ, seg_pos={0 0}, upload=463754]: Uploading piece... Dec 27 16:14:00.651738606 (528h45m0.522821651s) [noise=true , address=145.79.227.50:4373 , piece_id=6JTO6RGTU7Y5FCWTOCO3GETORO22Z6PGM45VB6XZYTK4SZSDVTBA, seg_pos={0 0}, upload=463754]: Uploading piece... Dec 27 16:14:00.652216946 (528h45m0.523299991s) [noise=true , address=45.91.116.9:4263 , piece_id=UPDWYF44SAQGKBHDP36VUX7L3JBL6Z4BDEPKFV24E2QMLSIGXVVA, seg_pos={0 0}, upload=463754]: Uploading piece... Dec 27 16:14:00.653950998 (528h45m0.525034033s) [noise=true , address=145.79.227.52:4446 , piece_id=ANW4AE5SGOR55XZXJ7Z6YAD57UCKIB7PF4MSDJHYPVKP53VWVYQA, seg_pos={0 0}, upload=463754]: Uploading piece... Dec 27 16:14:00.654889345 (528h45m0.52597239s) [noise=true , address=145.79.227.17:4971 , piece_id=WKHHC73S3PPMTVOG5V3AOMKRLU7MEQMTTP7ULW6NH3LQ6W6DCLGQ, seg_pos={0 0}, upload=463754]: Uploading piece... Dec 27 16:14:00.655971104 (528h45m0.527054149s) [noise=true , address=145.79.227.11:4731 , piece_id=6UYBMXU4N3JUVIIPVMALYUU7BUZ7JMO7WY6NXHGGEKOCCOVYSA5A, seg_pos={0 0}, upload=463754]: Uploading piece... Dec 27 16:14:02.355889878 (528h45m2.226972923s) [upload=463754]: Next returned nil segment Dec 27 16:14:02.355914986 (528h45m2.226998031s) [upload=463754]: Waiting for error group managing segments... Dec 27 16:14:02.640758077 (528h45m2.511841152s) [noise=true , address=145.79.227.37:4565 , piece_id=FQYIE47LGUVVODV7IXWAJDNIKFYGUP5AISJ2J7FRTMLS4LHX2P7A, seg_pos={0 0}, upload=463754]: Done uploading piece. err: <nil> Dec 27 16:14:02.641655366 (528h45m2.512738411s) [noise=true , address=145.79.227.24:4581 , piece_id=NEUMSOWQ6SBUG3KVCQ4AQUXCV62JVAWCX4RICUIU2C5KVPTDENXQ, seg_pos={0 0}, upload=463754]: Done uploading piece. err: <nil> Dec 27 16:14:02.661912469 (528h45m2.532995514s) [noise=true , address=145.79.227.4:4186 , piece_id=M3AZMAR6KQPV7S2MXAQUV7SNVTZAD3MP2F5JF55JMV6OGS6T6U5Q, seg_pos={0 0}, upload=463754]: Done uploading piece. err: <nil> Dec 27 16:14:02.662737941 (528h45m2.533820986s) [noise=true , address=145.79.227.27:4691 , piece_id=RALEIBLDQF7DDVJ63ROTSXYWIHJIEJHWGLCGI2S3Y5Q4YC7UYTDQ, seg_pos={0 0}, upload=463754]: Done uploading piece. err: <nil> Dec 27 16:14:02.665303425 (528h45m2.53638647s) [noise=true , address=145.79.227.50:4373 , piece_id=6JTO6RGTU7Y5FCWTOCO3GETORO22Z6PGM45VB6XZYTK4SZSDVTBA, seg_pos={0 0}, upload=463754]: Done uploading piece. err: <nil> Dec 27 16:14:02.669673527 (528h45m2.540756582s) [noise=true , address=145.79.227.52:4446 , piece_id=ANW4AE5SGOR55XZXJ7Z6YAD57UCKIB7PF4MSDJHYPVKP53VWVYQA, seg_pos={0 0}, upload=463754]: Done uploading piece. err: <nil> Dec 27 16:14:02.670652311 (528h45m2.541735356s) [noise=true , address=145.79.227.26:4655 , piece_id=MM4DZIMGAUECSLW37CCBS5ID7OO2637RJSX6DYKQSD6RDL6GZH7Q, seg_pos={0 0}, upload=463754]: Done uploading piece. err: <nil> Dec 27 16:14:02.672338923 (528h45m2.543421968s) [noise=true , address=145.79.227.10:4403 , piece_id=JECL6CBUJ5WEUXICHCLQ5SVQB6C276QD4ISOLZVNSKV3QF7RKP3Q, seg_pos={0 0}, upload=463754]: Done uploading piece. err: <nil> Dec 27 16:14:02.680083409 (528h45m2.551166444s) [noise=true , address=145.79.227.17:4971 , piece_id=WKHHC73S3PPMTVOG5V3AOMKRLU7MEQMTTP7ULW6NH3LQ6W6DCLGQ, seg_pos={0 0}, upload=463754]: Done uploading piece. err: <nil> Dec 27 16:14:02.686131456 (528h45m2.557214501s) [noise=true , address=145.79.227.11:4731 , piece_id=6UYBMXU4N3JUVIIPVMALYUU7BUZ7JMO7WY6NXHGGEKOCCOVYSA5A, seg_pos={0 0}, upload=463754]: Done uploading piece. err: <nil> Dec 27 16:14:02.694673852 (528h45m2.565756897s) [noise=true , address=145.79.227.51:4411 , piece_id=D5EJV7BL6KUDHSDE3MZJ6TG5LT4O5SOB62QO52YAA7VMH4HBLAKQ, seg_pos={0 0}, upload=463754]: Done uploading piece. err: <nil> Dec 27 16:14:02.911872668 (528h45m2.782955713s) [noise=true , address=77.90.31.11:4555 , piece_id=WJEEGB47JBD5XK7GGSXZOS3YMFGJLLYUT64LVYCYBYW273XE7MAQ, seg_pos={0 0}, upload=463754]: Done uploading piece. err: <nil> Dec 27 16:14:03.043646287 (528h45m2.914729332s) [noise=true , address=45.91.116.13:4153 , piece_id=BRM46MJ7BD4GU2QYFMAZJOZSU6OHYR4KNZOYPEM277ROQ4WRQTWA, seg_pos={0 0}, upload=463754]: Done uploading piece. err: <nil> Dec 27 16:14:03.043672297 (528h45m2.914755342s) [seg_pos={0 0}, upload=463754]: Segment reached optimal threshold of 13 pieces. Dec 27 16:14:03.044120831 (528h45m2.915203886s) [noise=true , address=45.91.116.6:4975 , piece_id=BIMPZTYTSQBHEWNMKQXI5LGD2K2DIJ2S64KW7WAD62YJZPTRQWOQ, seg_pos={0 0}, upload=463754]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:42102->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 16:14:03.044133815 (528h45m2.91521686s) [noise=true , address=45.91.116.6:4975 , piece_id=BIMPZTYTSQBHEWNMKQXI5LGD2K2DIJ2S64KW7WAD62YJZPTRQWOQ, seg_pos={0 0}, upload=463754]: Long tail cancellation due to optimal threshold Dec 27 16:14:03.044224889 (528h45m2.915307944s) [noise=true , address=45.91.116.4:4937 , piece_id=QX6L4SHXTY4ZUKWE6BNHQBBMBJCS5M4TZF5RW6L2POF35I2YNXJA, seg_pos={0 0}, upload=463754]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: read tcp 188.166.225.170:60248->45.91.116.4:4937: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 16:14:03.044234517 (528h45m2.915317562s) [noise=true , address=45.91.116.4:4937 , piece_id=QX6L4SHXTY4ZUKWE6BNHQBBMBJCS5M4TZF5RW6L2POF35I2YNXJA, seg_pos={0 0}, upload=463754]: Long tail cancellation due to optimal threshold Dec 27 16:14:03.044267720 (528h45m2.915350775s) [noise=true , address=45.91.116.9:4263 , piece_id=UPDWYF44SAQGKBHDP36VUX7L3JBL6Z4BDEPKFV24E2QMLSIGXVVA, seg_pos={0 0}, upload=463754]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: write tcp 188.166.225.170:59592->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:59592->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59592->45.91.116.9:4263: use of closed network connection Dec 27 16:14:03.044276807 (528h45m2.915359852s) [noise=true , address=45.91.116.9:4263 , piece_id=UPDWYF44SAQGKBHDP36VUX7L3JBL6Z4BDEPKFV24E2QMLSIGXVVA, seg_pos={0 0}, upload=463754]: Long tail cancellation due to optimal threshold Dec 27 16:14:03.044313397 (528h45m2.915396442s) [seg_pos={0 0}, upload=463754]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:14:03.044523146 (528h45m2.915606191s) [upload=463754]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:14:03.044532525 (528h45m2.91561557s) [upload=463754]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:14:03.341530340 (528h45m3.212613395s) [upload=463754]: Done uploading segments... Dec 27 16:14:03.341557071 (528h45m3.212640117s) [upload=463754]: Upload finished. err: <nil> Duration: 6.350904359s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..XX......X................XX...............................] duration: 47.140911369s trace: https://jaeger.dev.storj.io/trace/9fd07cafe8f934a FAIL (2025-12-27 16:14:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315411726 (528h16m20.186494771s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315905206 (528h16m20.186988251s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316356304 (528h16m20.187439339s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316793537 (528h16m20.187876582s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.317233315 (528h16m20.18831636s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318042255 (528h16m20.1891253s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318150922 (528h16m20.189233967s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318513712 (528h16m20.189596747s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318955464 (528h16m20.190038509s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319037570 (528h16m20.190120605s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319389541 (528h16m20.190472576s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319874033 (528h16m20.190957078s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320378844 (528h16m20.191461879s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320898604 (528h16m20.191981649s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.321434254 (528h16m20.192517289s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:21.620856000 (528h16m21.491939035s) [upload=463344]: Next returned nil segment Dec 27 15:45:21.620875698 (528h16m21.491958743s) [upload=463344]: Waiting for error group managing segments... Dec 27 15:45:21.953178480 (528h16m21.824261516s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.194632602 (528h16m22.065715647s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.195118968 (528h16m22.066202013s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.204573761 (528h16m22.075656806s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.213413343 (528h16m22.084496388s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.372654386 (528h16m22.243737431s) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.473126101 (528h16m22.344209146s) [noise=true , address=77.90.31.12:4615 , piece_id=NTYP2JTUJYSBE4NBMQHTEAY2DA333CR7V3622PVQ3OGRMSPCJK3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.484926481 (528h16m22.356009536s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.563981276 (528h16m22.435064311s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.893470090 (528h16m22.764553126s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.105921755 (528h16m22.9770048s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.153398930 (528h16m23.024481975s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.218328398 (528h16m23.089411443s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244693782 (528h16m23.115776827s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244714231 (528h16m23.115797266s) [seg_pos={0 0}, upload=463344]: Segment reached optimal threshold of 14 pieces. Dec 27 15:45:23.244907037 (528h16m23.115990073s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.244913159 (528h16m23.115996204s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245080929 (528h16m23.116163964s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245087220 (528h16m23.116170275s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245280880 (528h16m23.116363925s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection Dec 27 15:45:23.245370850 (528h16m23.116453885s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245453588 (528h16m23.116536644s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f2MspBPykwYCWAYzJ9PCAXop6mPNarJt3fa6fWZoLipZZQdy1): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245474488 (528h16m23.116557533s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245514535 (528h16m23.11659757s) [seg_pos={0 0}, upload=463344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:45:23.246027922 (528h16m23.117110977s) [upload=463344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:45:23.246044133 (528h16m23.117127168s) [upload=463344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:45:23.560988923 (528h16m23.432071968s) [upload=463344]: Done uploading segments... Dec 27 15:45:23.561009573 (528h16m23.432092618s) [upload=463344]: Upload finished. err: <nil> Duration: 47.140939011s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X.X..X..XXX...........X.......X..X.XX..X..X....XX.X....X..X] duration: 30.003564152s trace: https://jaeger.dev.storj.io/trace/5f1261b77569ffc1 FAIL (2025-12-27 16:14:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: WJEEGB47JBD5XK7GGSXZOS3YMFGJLLYUT64LVYCYBYW273XE7MAQ) context deadline exceeded; piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: BRM46MJ7BD4GU2QYFMAZJOZSU6OHYR4KNZOYPEM277ROQ4WRQTWA) context deadline exceeded ======LOGS====== upload Dec 27 16:14:00.401050972 (528h45m0.272134028s) [upload=463754]: Uploading segments... Dec 27 16:14:00.401161513 (528h45m0.272244558s) [upload=463754]: Got next segment. Inline: false Dec 27 16:14:00.401215916 (528h45m0.272298971s) [upload=463754]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:14:00.401223741 (528h45m0.272306807s) [upload=463754]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:14:00.648483336 (528h45m0.519566372s) [seg_pos={0 0}, upload=463754]: Begin upload segment... Dec 27 16:14:00.648541638 (528h45m0.519624673s) [seg_pos={0 0}, upload=463754]: Done begin upload segment. Dec 27 16:14:00.648595961 (528h45m0.519678996s) [noise=true , address=145.79.227.4:4186 , piece_id=M3AZMAR6KQPV7S2MXAQUV7SNVTZAD3MP2F5JF55JMV6OGS6T6U5Q, seg_pos={0 0}, upload=463754]: Uploading piece... Dec 27 16:14:00.648623573 (528h45m0.519706619s) [noise=true , address=145.79.227.24:4581 , piece_id=NEUMSOWQ6SBUG3KVCQ4AQUXCV62JVAWCX4RICUIU2C5KVPTDENXQ, seg_pos={0 0}, upload=463754]: Uploading piece... Dec 27 16:14:00.649330780 (528h45m0.520413815s) [noise=true , address=145.79.227.10:4403 , piece_id=JECL6CBUJ5WEUXICHCLQ5SVQB6C276QD4ISOLZVNSKV3QF7RKP3Q, seg_pos={0 0}, upload=463754]: Uploading piece... Dec 27 16:14:00.649490454 (528h45m0.520573499s) [noise=true , address=145.79.227.37:4565 , piece_id=FQYIE47LGUVVODV7IXWAJDNIKFYGUP5AISJ2J7FRTMLS4LHX2P7A, seg_pos={0 0}, upload=463754]: Uploading piece... Dec 27 16:14:00.649784244 (528h45m0.520867289s) [noise=true , address=77.90.31.11:4555 , piece_id=WJEEGB47JBD5XK7GGSXZOS3YMFGJLLYUT64LVYCYBYW273XE7MAQ, seg_pos={0 0}, upload=463754]: Uploading piece... Dec 27 16:14:00.650253597 (528h45m0.521336633s) [noise=true , address=145.79.227.26:4655 , piece_id=MM4DZIMGAUECSLW37CCBS5ID7OO2637RJSX6DYKQSD6RDL6GZH7Q, seg_pos={0 0}, upload=463754]: Uploading piece... Dec 27 16:14:00.650365070 (528h45m0.521448105s) [noise=true , address=145.79.227.51:4411 , piece_id=D5EJV7BL6KUDHSDE3MZJ6TG5LT4O5SOB62QO52YAA7VMH4HBLAKQ, seg_pos={0 0}, upload=463754]: Uploading piece... Dec 27 16:14:00.650814495 (528h45m0.52189754s) [noise=true , address=145.79.227.27:4691 , piece_id=RALEIBLDQF7DDVJ63ROTSXYWIHJIEJHWGLCGI2S3Y5Q4YC7UYTDQ, seg_pos={0 0}, upload=463754]: Uploading piece... Dec 27 16:14:00.650934795 (528h45m0.52201783s) [noise=true , address=45.91.116.4:4937 , piece_id=QX6L4SHXTY4ZUKWE6BNHQBBMBJCS5M4TZF5RW6L2POF35I2YNXJA, seg_pos={0 0}, upload=463754]: Uploading piece... Dec 27 16:14:00.651282788 (528h45m0.522365823s) [noise=true , address=45.91.116.13:4153 , piece_id=BRM46MJ7BD4GU2QYFMAZJOZSU6OHYR4KNZOYPEM277ROQ4WRQTWA, seg_pos={0 0}, upload=463754]: Uploading piece... Dec 27 16:14:00.651569283 (528h45m0.522652318s) [noise=true , address=45.91.116.6:4975 , piece_id=BIMPZTYTSQBHEWNMKQXI5LGD2K2DIJ2S64KW7WAD62YJZPTRQWOQ, seg_pos={0 0}, upload=463754]: Uploading piece... Dec 27 16:14:00.651738606 (528h45m0.522821651s) [noise=true , address=145.79.227.50:4373 , piece_id=6JTO6RGTU7Y5FCWTOCO3GETORO22Z6PGM45VB6XZYTK4SZSDVTBA, seg_pos={0 0}, upload=463754]: Uploading piece... Dec 27 16:14:00.652216946 (528h45m0.523299991s) [noise=true , address=45.91.116.9:4263 , piece_id=UPDWYF44SAQGKBHDP36VUX7L3JBL6Z4BDEPKFV24E2QMLSIGXVVA, seg_pos={0 0}, upload=463754]: Uploading piece... Dec 27 16:14:00.653950998 (528h45m0.525034033s) [noise=true , address=145.79.227.52:4446 , piece_id=ANW4AE5SGOR55XZXJ7Z6YAD57UCKIB7PF4MSDJHYPVKP53VWVYQA, seg_pos={0 0}, upload=463754]: Uploading piece... Dec 27 16:14:00.654889345 (528h45m0.52597239s) [noise=true , address=145.79.227.17:4971 , piece_id=WKHHC73S3PPMTVOG5V3AOMKRLU7MEQMTTP7ULW6NH3LQ6W6DCLGQ, seg_pos={0 0}, upload=463754]: Uploading piece... Dec 27 16:14:00.655971104 (528h45m0.527054149s) [noise=true , address=145.79.227.11:4731 , piece_id=6UYBMXU4N3JUVIIPVMALYUU7BUZ7JMO7WY6NXHGGEKOCCOVYSA5A, seg_pos={0 0}, upload=463754]: Uploading piece... Dec 27 16:14:02.355889878 (528h45m2.226972923s) [upload=463754]: Next returned nil segment Dec 27 16:14:02.355914986 (528h45m2.226998031s) [upload=463754]: Waiting for error group managing segments... Dec 27 16:14:02.640758077 (528h45m2.511841152s) [noise=true , address=145.79.227.37:4565 , piece_id=FQYIE47LGUVVODV7IXWAJDNIKFYGUP5AISJ2J7FRTMLS4LHX2P7A, seg_pos={0 0}, upload=463754]: Done uploading piece. err: <nil> Dec 27 16:14:02.641655366 (528h45m2.512738411s) [noise=true , address=145.79.227.24:4581 , piece_id=NEUMSOWQ6SBUG3KVCQ4AQUXCV62JVAWCX4RICUIU2C5KVPTDENXQ, seg_pos={0 0}, upload=463754]: Done uploading piece. err: <nil> Dec 27 16:14:02.661912469 (528h45m2.532995514s) [noise=true , address=145.79.227.4:4186 , piece_id=M3AZMAR6KQPV7S2MXAQUV7SNVTZAD3MP2F5JF55JMV6OGS6T6U5Q, seg_pos={0 0}, upload=463754]: Done uploading piece. err: <nil> Dec 27 16:14:02.662737941 (528h45m2.533820986s) [noise=true , address=145.79.227.27:4691 , piece_id=RALEIBLDQF7DDVJ63ROTSXYWIHJIEJHWGLCGI2S3Y5Q4YC7UYTDQ, seg_pos={0 0}, upload=463754]: Done uploading piece. err: <nil> Dec 27 16:14:02.665303425 (528h45m2.53638647s) [noise=true , address=145.79.227.50:4373 , piece_id=6JTO6RGTU7Y5FCWTOCO3GETORO22Z6PGM45VB6XZYTK4SZSDVTBA, seg_pos={0 0}, upload=463754]: Done uploading piece. err: <nil> Dec 27 16:14:02.669673527 (528h45m2.540756582s) [noise=true , address=145.79.227.52:4446 , piece_id=ANW4AE5SGOR55XZXJ7Z6YAD57UCKIB7PF4MSDJHYPVKP53VWVYQA, seg_pos={0 0}, upload=463754]: Done uploading piece. err: <nil> Dec 27 16:14:02.670652311 (528h45m2.541735356s) [noise=true , address=145.79.227.26:4655 , piece_id=MM4DZIMGAUECSLW37CCBS5ID7OO2637RJSX6DYKQSD6RDL6GZH7Q, seg_pos={0 0}, upload=463754]: Done uploading piece. err: <nil> Dec 27 16:14:02.672338923 (528h45m2.543421968s) [noise=true , address=145.79.227.10:4403 , piece_id=JECL6CBUJ5WEUXICHCLQ5SVQB6C276QD4ISOLZVNSKV3QF7RKP3Q, seg_pos={0 0}, upload=463754]: Done uploading piece. err: <nil> Dec 27 16:14:02.680083409 (528h45m2.551166444s) [noise=true , address=145.79.227.17:4971 , piece_id=WKHHC73S3PPMTVOG5V3AOMKRLU7MEQMTTP7ULW6NH3LQ6W6DCLGQ, seg_pos={0 0}, upload=463754]: Done uploading piece. err: <nil> Dec 27 16:14:02.686131456 (528h45m2.557214501s) [noise=true , address=145.79.227.11:4731 , piece_id=6UYBMXU4N3JUVIIPVMALYUU7BUZ7JMO7WY6NXHGGEKOCCOVYSA5A, seg_pos={0 0}, upload=463754]: Done uploading piece. err: <nil> Dec 27 16:14:02.694673852 (528h45m2.565756897s) [noise=true , address=145.79.227.51:4411 , piece_id=D5EJV7BL6KUDHSDE3MZJ6TG5LT4O5SOB62QO52YAA7VMH4HBLAKQ, seg_pos={0 0}, upload=463754]: Done uploading piece. err: <nil> Dec 27 16:14:02.911872668 (528h45m2.782955713s) [noise=true , address=77.90.31.11:4555 , piece_id=WJEEGB47JBD5XK7GGSXZOS3YMFGJLLYUT64LVYCYBYW273XE7MAQ, seg_pos={0 0}, upload=463754]: Done uploading piece. err: <nil> Dec 27 16:14:03.043646287 (528h45m2.914729332s) [noise=true , address=45.91.116.13:4153 , piece_id=BRM46MJ7BD4GU2QYFMAZJOZSU6OHYR4KNZOYPEM277ROQ4WRQTWA, seg_pos={0 0}, upload=463754]: Done uploading piece. err: <nil> Dec 27 16:14:03.043672297 (528h45m2.914755342s) [seg_pos={0 0}, upload=463754]: Segment reached optimal threshold of 13 pieces. Dec 27 16:14:03.044120831 (528h45m2.915203886s) [noise=true , address=45.91.116.6:4975 , piece_id=BIMPZTYTSQBHEWNMKQXI5LGD2K2DIJ2S64KW7WAD62YJZPTRQWOQ, seg_pos={0 0}, upload=463754]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: read tcp 188.166.225.170:42102->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 16:14:03.044133815 (528h45m2.91521686s) [noise=true , address=45.91.116.6:4975 , piece_id=BIMPZTYTSQBHEWNMKQXI5LGD2K2DIJ2S64KW7WAD62YJZPTRQWOQ, seg_pos={0 0}, upload=463754]: Long tail cancellation due to optimal threshold Dec 27 16:14:03.044224889 (528h45m2.915307944s) [noise=true , address=45.91.116.4:4937 , piece_id=QX6L4SHXTY4ZUKWE6BNHQBBMBJCS5M4TZF5RW6L2POF35I2YNXJA, seg_pos={0 0}, upload=463754]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: read tcp 188.166.225.170:60248->45.91.116.4:4937: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 16:14:03.044234517 (528h45m2.915317562s) [noise=true , address=45.91.116.4:4937 , piece_id=QX6L4SHXTY4ZUKWE6BNHQBBMBJCS5M4TZF5RW6L2POF35I2YNXJA, seg_pos={0 0}, upload=463754]: Long tail cancellation due to optimal threshold Dec 27 16:14:03.044267720 (528h45m2.915350775s) [noise=true , address=45.91.116.9:4263 , piece_id=UPDWYF44SAQGKBHDP36VUX7L3JBL6Z4BDEPKFV24E2QMLSIGXVVA, seg_pos={0 0}, upload=463754]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: write tcp 188.166.225.170:59592->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:59592->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59592->45.91.116.9:4263: use of closed network connection Dec 27 16:14:03.044276807 (528h45m2.915359852s) [noise=true , address=45.91.116.9:4263 , piece_id=UPDWYF44SAQGKBHDP36VUX7L3JBL6Z4BDEPKFV24E2QMLSIGXVVA, seg_pos={0 0}, upload=463754]: Long tail cancellation due to optimal threshold Dec 27 16:14:03.044313397 (528h45m2.915396442s) [seg_pos={0 0}, upload=463754]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:14:03.044523146 (528h45m2.915606191s) [upload=463754]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:14:03.044532525 (528h45m2.91561557s) [upload=463754]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:14:03.341530340 (528h45m3.212613395s) [upload=463754]: Done uploading segments... Dec 27 16:14:03.341557071 (528h45m3.212640117s) [upload=463754]: Upload finished. err: <nil> Duration: 30.003589511s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [...X...XX.XXXX....XXXX..XX..XX......X......XX...X...........] duration: 5.00086287s trace: https://jaeger.dev.storj.io/trace/36696332d68ad959 FAIL (2025-12-27 16:14:14 UTC) orbiter warning: context deadline exceeded Duration: 5.00088853s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...XX......X................XX..............................] duration: 52.004463728s trace: https://jaeger.dev.storj.io/trace/9fd07cafe8f934a FAIL (2025-12-27 16:13:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315411726 (528h16m20.186494771s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315905206 (528h16m20.186988251s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316356304 (528h16m20.187439339s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316793537 (528h16m20.187876582s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.317233315 (528h16m20.18831636s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318042255 (528h16m20.1891253s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318150922 (528h16m20.189233967s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318513712 (528h16m20.189596747s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318955464 (528h16m20.190038509s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319037570 (528h16m20.190120605s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319389541 (528h16m20.190472576s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319874033 (528h16m20.190957078s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320378844 (528h16m20.191461879s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320898604 (528h16m20.191981649s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.321434254 (528h16m20.192517289s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:21.620856000 (528h16m21.491939035s) [upload=463344]: Next returned nil segment Dec 27 15:45:21.620875698 (528h16m21.491958743s) [upload=463344]: Waiting for error group managing segments... Dec 27 15:45:21.953178480 (528h16m21.824261516s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.194632602 (528h16m22.065715647s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.195118968 (528h16m22.066202013s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.204573761 (528h16m22.075656806s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.213413343 (528h16m22.084496388s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.372654386 (528h16m22.243737431s) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.473126101 (528h16m22.344209146s) [noise=true , address=77.90.31.12:4615 , piece_id=NTYP2JTUJYSBE4NBMQHTEAY2DA333CR7V3622PVQ3OGRMSPCJK3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.484926481 (528h16m22.356009536s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.563981276 (528h16m22.435064311s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.893470090 (528h16m22.764553126s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.105921755 (528h16m22.9770048s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.153398930 (528h16m23.024481975s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.218328398 (528h16m23.089411443s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244693782 (528h16m23.115776827s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244714231 (528h16m23.115797266s) [seg_pos={0 0}, upload=463344]: Segment reached optimal threshold of 14 pieces. Dec 27 15:45:23.244907037 (528h16m23.115990073s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.244913159 (528h16m23.115996204s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245080929 (528h16m23.116163964s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245087220 (528h16m23.116170275s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245280880 (528h16m23.116363925s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection Dec 27 15:45:23.245370850 (528h16m23.116453885s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245453588 (528h16m23.116536644s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f2MspBPykwYCWAYzJ9PCAXop6mPNarJt3fa6fWZoLipZZQdy1): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245474488 (528h16m23.116557533s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245514535 (528h16m23.11659757s) [seg_pos={0 0}, upload=463344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:45:23.246027922 (528h16m23.117110977s) [upload=463344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:45:23.246044133 (528h16m23.117127168s) [upload=463344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:45:23.560988923 (528h16m23.432071968s) [upload=463344]: Done uploading segments... Dec 27 15:45:23.561009573 (528h16m23.432092618s) [upload=463344]: Upload finished. err: <nil> Duration: 52.004491039s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [....X...XX.XXXX....XXXX..XX..XX......X......XX...X..........] duration: 3.534160232s trace: https://jaeger.dev.storj.io/trace/36696332d68ad959 FAIL (2025-12-27 16:13:31 UTC) orbiter warning: context deadline exceeded Duration: 3.534177946s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X.X..X..XXX...........X.......X..X.XX..X..X....XX.X....X..] duration: 5.895816521s trace: https://jaeger.dev.storj.io/trace/7fd0ee7047f5c08e FAIL (2025-12-27 16:12:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: O42ZCYNIMYDWTOHMWJVOA6FV4SSK2E4CFNV3H6VXYXWO72IARYLQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: IHJZGTHXEFDYNUKE2CMSZKLSIW5LTMDMLUP63CWGNTMTHGHYVP6Q) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: CYNBDIE6ICWWXQDGFE4X32ACKUNUZ3XENW7ARAD23UNANRLT5VFA) context deadline exceeded ======LOGS====== Dec 27 16:10:56.181719477 (528h41m56.052802522s) [upload=463709]: Starting upload Dec 27 16:10:56.181930308 (528h41m56.053013353s) [upload=463709]: Done starting upload Dec 27 16:10:56.182030688 (528h41m56.053113723s) [upload=463709]: Uploading segments... Dec 27 16:10:56.182119939 (528h41m56.053202984s) [upload=463709]: Got next segment. Inline: false Dec 27 16:10:56.182176777 (528h41m56.053259822s) [upload=463709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:10:56.182183340 (528h41m56.053266385s) [upload=463709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:10:56.423123263 (528h41m56.294206308s) [seg_pos={0 0}, upload=463709]: Begin upload segment... Dec 27 16:10:56.423180442 (528h41m56.294263477s) [seg_pos={0 0}, upload=463709]: Done begin upload segment. Dec 27 16:10:56.423215959 (528h41m56.294298994s) [noise=true , address=77.90.31.21:4877 , piece_id=O42ZCYNIMYDWTOHMWJVOA6FV4SSK2E4CFNV3H6VXYXWO72IARYLQ, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.423229645 (528h41m56.29431269s) [noise=true , address=45.91.116.7:4012 , piece_id=L5245B7VAWC4OFEXVUVTAO5QTCWFHBXM6IMZDJZ4KYJQOYPPYGOA, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.423670715 (528h41m56.29475375s) [noise=true , address=145.79.227.48:4301 , piece_id=LPSMU4EFWM2LUCUZWYNHVC3Q22DJG2BHZBAXNONQBP6463ZXBM2Q, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.424568335 (528h41m56.29565138s) [noise=true , address=45.91.116.13:4156 , piece_id=LKRWLMDO7WWOIXQNQEZBHSURCURKET7FJ2Q5BPCRKFK6UMSSWOJA, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.424862064 (528h41m56.295945109s) [noise=true , address=77.90.31.19:4844 , piece_id=ZZIHQCH6BYGR4ICUYEWEMSG7CHEHXZFI5UZ5J4CW2ZK5O4JTJ67Q, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.425136466 (528h41m56.296219511s) [noise=true , address=45.91.116.14:4190 , piece_id=CYNBDIE6ICWWXQDGFE4X32ACKUNUZ3XENW7ARAD23UNANRLT5VFA, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.425497003 (528h41m56.296580038s) [noise=true , address=145.79.227.5:4223 , piece_id=CABTYGFXJNPQHTNN2EVYEJQP4Y3ONH3UGHGU2I73U73EHUKR5NIQ, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.425999510 (528h41m56.297082545s) [noise=true , address=145.79.227.37:4566 , piece_id=5FFP6FG6EWG5AGDD22FJ3ZVXU5LK3DCZASZUIMPMDXYNA7HYKUYA, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.426379493 (528h41m56.297462528s) [noise=true , address=145.79.227.16:4912 , piece_id=DLYJQNA3Z7SRLFOI4ZXNVBE5WPRI27TH3CVY4VZSQ2IEXDQJKHNQ, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.427552357 (528h41m56.298635402s) [noise=true , address=145.79.227.36:4519 , piece_id=SPQFN365JO3KCD22GO3JP6QXKXHMRINNIVV4S7WBE4Q5EJQUZWNA, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.428678170 (528h41m56.299761216s) [noise=true , address=145.79.227.51:4411 , piece_id=6SAIOEKXELQAI6ESPRGEAJ46S4KKUCVD3ZUEVBIJ5MWVCWSF6MZA, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.429903864 (528h41m56.300986899s) [noise=true , address=77.90.31.16:4736 , piece_id=IHJZGTHXEFDYNUKE2CMSZKLSIW5LTMDMLUP63CWGNTMTHGHYVP6Q, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.434390167 (528h41m56.305473202s) [noise=true , address=77.90.31.15:4697 , piece_id=R4I5W5DSS6R5AQCVAQYSIBAR4OMYSAAHYYZRKHWPA5NL62MRP4NA, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.438281928 (528h41m56.309364973s) [noise=true , address=145.79.227.15:4876 , piece_id=RYZDEKIZ7J42DLSUN4JMQVITHBAIAOZZLNOD3T75DTBWO7QBYLZQ, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.440522083 (528h41m56.311605118s) [noise=true , address=77.90.31.10:4519 , piece_id=ZZGKFH6GJS3QZNRXUEIK5CUZ6IQYZ6EDFOJS6WUWPGFEBN5UTX6Q, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.442575884 (528h41m56.313658929s) [noise=true , address=145.79.227.47:4265 , piece_id=6KKF6M25OJQXKBRP2EJWYASMDNV57RO63NENZ3HXMUEWZPZL4FGA, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:58.059945125 (528h41m57.93102817s) [upload=463709]: Next returned nil segment Dec 27 16:10:58.059964573 (528h41m57.931047638s) [upload=463709]: Waiting for error group managing segments... Dec 27 16:10:58.442778428 (528h41m58.313861473s) [noise=true , address=145.79.227.15:4876 , piece_id=RYZDEKIZ7J42DLSUN4JMQVITHBAIAOZZLNOD3T75DTBWO7QBYLZQ, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.444001778 (528h41m58.315084813s) [noise=true , address=145.79.227.37:4566 , piece_id=5FFP6FG6EWG5AGDD22FJ3ZVXU5LK3DCZASZUIMPMDXYNA7HYKUYA, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.444087872 (528h41m58.315170917s) [noise=true , address=145.79.227.48:4301 , piece_id=LPSMU4EFWM2LUCUZWYNHVC3Q22DJG2BHZBAXNONQBP6463ZXBM2Q, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.444465411 (528h41m58.315548456s) [noise=true , address=145.79.227.5:4223 , piece_id=CABTYGFXJNPQHTNN2EVYEJQP4Y3ONH3UGHGU2I73U73EHUKR5NIQ, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.450753845 (528h41m58.32183688s) [noise=true , address=145.79.227.16:4912 , piece_id=DLYJQNA3Z7SRLFOI4ZXNVBE5WPRI27TH3CVY4VZSQ2IEXDQJKHNQ, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.456006076 (528h41m58.327089121s) [noise=true , address=145.79.227.51:4411 , piece_id=6SAIOEKXELQAI6ESPRGEAJ46S4KKUCVD3ZUEVBIJ5MWVCWSF6MZA, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.456607192 (528h41m58.327690227s) [noise=true , address=145.79.227.36:4519 , piece_id=SPQFN365JO3KCD22GO3JP6QXKXHMRINNIVV4S7WBE4Q5EJQUZWNA, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.477440491 (528h41m58.348523536s) [noise=true , address=145.79.227.47:4265 , piece_id=6KKF6M25OJQXKBRP2EJWYASMDNV57RO63NENZ3HXMUEWZPZL4FGA, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.637293700 (528h41m58.508376745s) [noise=true , address=77.90.31.16:4736 , piece_id=IHJZGTHXEFDYNUKE2CMSZKLSIW5LTMDMLUP63CWGNTMTHGHYVP6Q, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.670869946 (528h41m58.541952981s) [noise=true , address=77.90.31.21:4877 , piece_id=O42ZCYNIMYDWTOHMWJVOA6FV4SSK2E4CFNV3H6VXYXWO72IARYLQ, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.676529733 (528h41m58.547612768s) [noise=true , address=45.91.116.7:4012 , piece_id=L5245B7VAWC4OFEXVUVTAO5QTCWFHBXM6IMZDJZ4KYJQOYPPYGOA, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.724049649 (528h41m58.595132694s) [noise=true , address=77.90.31.10:4519 , piece_id=ZZGKFH6GJS3QZNRXUEIK5CUZ6IQYZ6EDFOJS6WUWPGFEBN5UTX6Q, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.741523653 (528h41m58.612606698s) [noise=true , address=45.91.116.14:4190 , piece_id=CYNBDIE6ICWWXQDGFE4X32ACKUNUZ3XENW7ARAD23UNANRLT5VFA, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.741545114 (528h41m58.612628159s) [seg_pos={0 0}, upload=463709]: Segment reached optimal threshold of 13 pieces. Dec 27 16:10:58.741915188 (528h41m58.612998243s) [noise=true , address=77.90.31.19:4844 , piece_id=ZZIHQCH6BYGR4ICUYEWEMSG7CHEHXZFI5UZ5J4CW2ZK5O4JTJ67Q, seg_pos={0 0}, upload=463709]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:57060->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:57060->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57060->77.90.31.19:4844: use of closed network connection Dec 27 16:10:58.741923425 (528h41m58.61300646s) [noise=true , address=77.90.31.19:4844 , piece_id=ZZIHQCH6BYGR4ICUYEWEMSG7CHEHXZFI5UZ5J4CW2ZK5O4JTJ67Q, seg_pos={0 0}, upload=463709]: Long tail cancellation due to optimal threshold Dec 27 16:10:58.742288279 (528h41m58.613371314s) [noise=true , address=45.91.116.13:4156 , piece_id=LKRWLMDO7WWOIXQNQEZBHSURCURKET7FJ2Q5BPCRKFK6UMSSWOJA, seg_pos={0 0}, upload=463709]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 16:10:58.742296074 (528h41m58.613379109s) [noise=true , address=45.91.116.13:4156 , piece_id=LKRWLMDO7WWOIXQNQEZBHSURCURKET7FJ2Q5BPCRKFK6UMSSWOJA, seg_pos={0 0}, upload=463709]: Long tail cancellation due to optimal threshold Dec 27 16:10:58.742469874 (528h41m58.61355292s) [noise=true , address=77.90.31.15:4697 , piece_id=R4I5W5DSS6R5AQCVAQYSIBAR4OMYSAAHYYZRKHWPA5NL62MRP4NA, seg_pos={0 0}, upload=463709]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 27 16:10:58.742477139 (528h41m58.613560184s) [noise=true , address=77.90.31.15:4697 , piece_id=R4I5W5DSS6R5AQCVAQYSIBAR4OMYSAAHYYZRKHWPA5NL62MRP4NA, seg_pos={0 0}, upload=463709]: Long tail cancellation due to optimal threshold Dec 27 16:10:58.742495514 (528h41m58.613578559s) [seg_pos={0 0}, upload=463709]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:10:58.742569665 (528h41m58.61365271s) [upload=463709]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:10:58.742575325 (528h41m58.61365837s) [upload=463709]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:10:59.032798984 (528h41m58.903882019s) [upload=463709]: Done uploading segments... Dec 27 16:10:59.032833740 (528h41m58.903916785s) [upload=463709]: Upload finished. err: <nil> Duration: 5.895845044s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [....XX......X................XX.............................] duration: 34.986163357s trace: https://jaeger.dev.storj.io/trace/9fd07cafe8f934a FAIL (2025-12-27 16:12:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315411726 (528h16m20.186494771s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315905206 (528h16m20.186988251s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316356304 (528h16m20.187439339s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316793537 (528h16m20.187876582s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.317233315 (528h16m20.18831636s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318042255 (528h16m20.1891253s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318150922 (528h16m20.189233967s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318513712 (528h16m20.189596747s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318955464 (528h16m20.190038509s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319037570 (528h16m20.190120605s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319389541 (528h16m20.190472576s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319874033 (528h16m20.190957078s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320378844 (528h16m20.191461879s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320898604 (528h16m20.191981649s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.321434254 (528h16m20.192517289s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:21.620856000 (528h16m21.491939035s) [upload=463344]: Next returned nil segment Dec 27 15:45:21.620875698 (528h16m21.491958743s) [upload=463344]: Waiting for error group managing segments... Dec 27 15:45:21.953178480 (528h16m21.824261516s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.194632602 (528h16m22.065715647s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.195118968 (528h16m22.066202013s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.204573761 (528h16m22.075656806s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.213413343 (528h16m22.084496388s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.372654386 (528h16m22.243737431s) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.473126101 (528h16m22.344209146s) [noise=true , address=77.90.31.12:4615 , piece_id=NTYP2JTUJYSBE4NBMQHTEAY2DA333CR7V3622PVQ3OGRMSPCJK3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.484926481 (528h16m22.356009536s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.563981276 (528h16m22.435064311s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.893470090 (528h16m22.764553126s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.105921755 (528h16m22.9770048s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.153398930 (528h16m23.024481975s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.218328398 (528h16m23.089411443s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244693782 (528h16m23.115776827s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244714231 (528h16m23.115797266s) [seg_pos={0 0}, upload=463344]: Segment reached optimal threshold of 14 pieces. Dec 27 15:45:23.244907037 (528h16m23.115990073s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.244913159 (528h16m23.115996204s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245080929 (528h16m23.116163964s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245087220 (528h16m23.116170275s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245280880 (528h16m23.116363925s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection Dec 27 15:45:23.245370850 (528h16m23.116453885s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245453588 (528h16m23.116536644s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f2MspBPykwYCWAYzJ9PCAXop6mPNarJt3fa6fWZoLipZZQdy1): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245474488 (528h16m23.116557533s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245514535 (528h16m23.11659757s) [seg_pos={0 0}, upload=463344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:45:23.246027922 (528h16m23.117110977s) [upload=463344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:45:23.246044133 (528h16m23.117127168s) [upload=463344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:45:23.560988923 (528h16m23.432071968s) [upload=463344]: Done uploading segments... Dec 27 15:45:23.561009573 (528h16m23.432092618s) [upload=463344]: Upload finished. err: <nil> Duration: 34.986188946s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [.....X...XX.XXXX....XXXX..XX..XX......X......XX...X.........] duration: 3.996036745s trace: https://jaeger.dev.storj.io/trace/36696332d68ad959 FAIL (2025-12-27 16:12:32 UTC) orbiter warning: context deadline exceeded Duration: 3.996059779s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [X.....X...XX.XXXX....XXXX..XX..XX......X......XX...X........] duration: 3.726496584s trace: https://jaeger.dev.storj.io/trace/36696332d68ad959 FAIL (2025-12-27 16:12:14 UTC) orbiter warning: context deadline exceeded Duration: 3.726523866s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....XX......X................XX............................] duration: 49.861114241s trace: https://jaeger.dev.storj.io/trace/9fd07cafe8f934a FAIL (2025-12-27 16:12:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315411726 (528h16m20.186494771s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315905206 (528h16m20.186988251s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316356304 (528h16m20.187439339s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316793537 (528h16m20.187876582s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.317233315 (528h16m20.18831636s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318042255 (528h16m20.1891253s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318150922 (528h16m20.189233967s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318513712 (528h16m20.189596747s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318955464 (528h16m20.190038509s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319037570 (528h16m20.190120605s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319389541 (528h16m20.190472576s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319874033 (528h16m20.190957078s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320378844 (528h16m20.191461879s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320898604 (528h16m20.191981649s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.321434254 (528h16m20.192517289s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:21.620856000 (528h16m21.491939035s) [upload=463344]: Next returned nil segment Dec 27 15:45:21.620875698 (528h16m21.491958743s) [upload=463344]: Waiting for error group managing segments... Dec 27 15:45:21.953178480 (528h16m21.824261516s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.194632602 (528h16m22.065715647s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.195118968 (528h16m22.066202013s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.204573761 (528h16m22.075656806s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.213413343 (528h16m22.084496388s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.372654386 (528h16m22.243737431s) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.473126101 (528h16m22.344209146s) [noise=true , address=77.90.31.12:4615 , piece_id=NTYP2JTUJYSBE4NBMQHTEAY2DA333CR7V3622PVQ3OGRMSPCJK3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.484926481 (528h16m22.356009536s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.563981276 (528h16m22.435064311s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.893470090 (528h16m22.764553126s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.105921755 (528h16m22.9770048s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.153398930 (528h16m23.024481975s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.218328398 (528h16m23.089411443s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244693782 (528h16m23.115776827s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244714231 (528h16m23.115797266s) [seg_pos={0 0}, upload=463344]: Segment reached optimal threshold of 14 pieces. Dec 27 15:45:23.244907037 (528h16m23.115990073s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.244913159 (528h16m23.115996204s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245080929 (528h16m23.116163964s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245087220 (528h16m23.116170275s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245280880 (528h16m23.116363925s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection Dec 27 15:45:23.245370850 (528h16m23.116453885s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245453588 (528h16m23.116536644s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f2MspBPykwYCWAYzJ9PCAXop6mPNarJt3fa6fWZoLipZZQdy1): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245474488 (528h16m23.116557533s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245514535 (528h16m23.11659757s) [seg_pos={0 0}, upload=463344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:45:23.246027922 (528h16m23.117110977s) [upload=463344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:45:23.246044133 (528h16m23.117127168s) [upload=463344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:45:23.560988923 (528h16m23.432071968s) [upload=463344]: Done uploading segments... Dec 27 15:45:23.561009573 (528h16m23.432092618s) [upload=463344]: Upload finished. err: <nil> Duration: 49.861132055s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...X.X..X..XXX...........X.......X..X.XX..X..X....XX.X....X.] duration: 6.380394719s trace: https://jaeger.dev.storj.io/trace/7fd0ee7047f5c08e FAIL (2025-12-27 16:11:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: O42ZCYNIMYDWTOHMWJVOA6FV4SSK2E4CFNV3H6VXYXWO72IARYLQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: IHJZGTHXEFDYNUKE2CMSZKLSIW5LTMDMLUP63CWGNTMTHGHYVP6Q) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: CYNBDIE6ICWWXQDGFE4X32ACKUNUZ3XENW7ARAD23UNANRLT5VFA) context deadline exceeded ======LOGS====== Dec 27 16:10:56.181719477 (528h41m56.052802522s) [upload=463709]: Starting upload Dec 27 16:10:56.181930308 (528h41m56.053013353s) [upload=463709]: Done starting upload Dec 27 16:10:56.182030688 (528h41m56.053113723s) [upload=463709]: Uploading segments... Dec 27 16:10:56.182119939 (528h41m56.053202984s) [upload=463709]: Got next segment. Inline: false Dec 27 16:10:56.182176777 (528h41m56.053259822s) [upload=463709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:10:56.182183340 (528h41m56.053266385s) [upload=463709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:10:56.423123263 (528h41m56.294206308s) [seg_pos={0 0}, upload=463709]: Begin upload segment... Dec 27 16:10:56.423180442 (528h41m56.294263477s) [seg_pos={0 0}, upload=463709]: Done begin upload segment. Dec 27 16:10:56.423215959 (528h41m56.294298994s) [noise=true , address=77.90.31.21:4877 , piece_id=O42ZCYNIMYDWTOHMWJVOA6FV4SSK2E4CFNV3H6VXYXWO72IARYLQ, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.423229645 (528h41m56.29431269s) [noise=true , address=45.91.116.7:4012 , piece_id=L5245B7VAWC4OFEXVUVTAO5QTCWFHBXM6IMZDJZ4KYJQOYPPYGOA, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.423670715 (528h41m56.29475375s) [noise=true , address=145.79.227.48:4301 , piece_id=LPSMU4EFWM2LUCUZWYNHVC3Q22DJG2BHZBAXNONQBP6463ZXBM2Q, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.424568335 (528h41m56.29565138s) [noise=true , address=45.91.116.13:4156 , piece_id=LKRWLMDO7WWOIXQNQEZBHSURCURKET7FJ2Q5BPCRKFK6UMSSWOJA, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.424862064 (528h41m56.295945109s) [noise=true , address=77.90.31.19:4844 , piece_id=ZZIHQCH6BYGR4ICUYEWEMSG7CHEHXZFI5UZ5J4CW2ZK5O4JTJ67Q, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.425136466 (528h41m56.296219511s) [noise=true , address=45.91.116.14:4190 , piece_id=CYNBDIE6ICWWXQDGFE4X32ACKUNUZ3XENW7ARAD23UNANRLT5VFA, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.425497003 (528h41m56.296580038s) [noise=true , address=145.79.227.5:4223 , piece_id=CABTYGFXJNPQHTNN2EVYEJQP4Y3ONH3UGHGU2I73U73EHUKR5NIQ, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.425999510 (528h41m56.297082545s) [noise=true , address=145.79.227.37:4566 , piece_id=5FFP6FG6EWG5AGDD22FJ3ZVXU5LK3DCZASZUIMPMDXYNA7HYKUYA, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.426379493 (528h41m56.297462528s) [noise=true , address=145.79.227.16:4912 , piece_id=DLYJQNA3Z7SRLFOI4ZXNVBE5WPRI27TH3CVY4VZSQ2IEXDQJKHNQ, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.427552357 (528h41m56.298635402s) [noise=true , address=145.79.227.36:4519 , piece_id=SPQFN365JO3KCD22GO3JP6QXKXHMRINNIVV4S7WBE4Q5EJQUZWNA, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.428678170 (528h41m56.299761216s) [noise=true , address=145.79.227.51:4411 , piece_id=6SAIOEKXELQAI6ESPRGEAJ46S4KKUCVD3ZUEVBIJ5MWVCWSF6MZA, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.429903864 (528h41m56.300986899s) [noise=true , address=77.90.31.16:4736 , piece_id=IHJZGTHXEFDYNUKE2CMSZKLSIW5LTMDMLUP63CWGNTMTHGHYVP6Q, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.434390167 (528h41m56.305473202s) [noise=true , address=77.90.31.15:4697 , piece_id=R4I5W5DSS6R5AQCVAQYSIBAR4OMYSAAHYYZRKHWPA5NL62MRP4NA, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.438281928 (528h41m56.309364973s) [noise=true , address=145.79.227.15:4876 , piece_id=RYZDEKIZ7J42DLSUN4JMQVITHBAIAOZZLNOD3T75DTBWO7QBYLZQ, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.440522083 (528h41m56.311605118s) [noise=true , address=77.90.31.10:4519 , piece_id=ZZGKFH6GJS3QZNRXUEIK5CUZ6IQYZ6EDFOJS6WUWPGFEBN5UTX6Q, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.442575884 (528h41m56.313658929s) [noise=true , address=145.79.227.47:4265 , piece_id=6KKF6M25OJQXKBRP2EJWYASMDNV57RO63NENZ3HXMUEWZPZL4FGA, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:58.059945125 (528h41m57.93102817s) [upload=463709]: Next returned nil segment Dec 27 16:10:58.059964573 (528h41m57.931047638s) [upload=463709]: Waiting for error group managing segments... Dec 27 16:10:58.442778428 (528h41m58.313861473s) [noise=true , address=145.79.227.15:4876 , piece_id=RYZDEKIZ7J42DLSUN4JMQVITHBAIAOZZLNOD3T75DTBWO7QBYLZQ, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.444001778 (528h41m58.315084813s) [noise=true , address=145.79.227.37:4566 , piece_id=5FFP6FG6EWG5AGDD22FJ3ZVXU5LK3DCZASZUIMPMDXYNA7HYKUYA, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.444087872 (528h41m58.315170917s) [noise=true , address=145.79.227.48:4301 , piece_id=LPSMU4EFWM2LUCUZWYNHVC3Q22DJG2BHZBAXNONQBP6463ZXBM2Q, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.444465411 (528h41m58.315548456s) [noise=true , address=145.79.227.5:4223 , piece_id=CABTYGFXJNPQHTNN2EVYEJQP4Y3ONH3UGHGU2I73U73EHUKR5NIQ, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.450753845 (528h41m58.32183688s) [noise=true , address=145.79.227.16:4912 , piece_id=DLYJQNA3Z7SRLFOI4ZXNVBE5WPRI27TH3CVY4VZSQ2IEXDQJKHNQ, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.456006076 (528h41m58.327089121s) [noise=true , address=145.79.227.51:4411 , piece_id=6SAIOEKXELQAI6ESPRGEAJ46S4KKUCVD3ZUEVBIJ5MWVCWSF6MZA, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.456607192 (528h41m58.327690227s) [noise=true , address=145.79.227.36:4519 , piece_id=SPQFN365JO3KCD22GO3JP6QXKXHMRINNIVV4S7WBE4Q5EJQUZWNA, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.477440491 (528h41m58.348523536s) [noise=true , address=145.79.227.47:4265 , piece_id=6KKF6M25OJQXKBRP2EJWYASMDNV57RO63NENZ3HXMUEWZPZL4FGA, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.637293700 (528h41m58.508376745s) [noise=true , address=77.90.31.16:4736 , piece_id=IHJZGTHXEFDYNUKE2CMSZKLSIW5LTMDMLUP63CWGNTMTHGHYVP6Q, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.670869946 (528h41m58.541952981s) [noise=true , address=77.90.31.21:4877 , piece_id=O42ZCYNIMYDWTOHMWJVOA6FV4SSK2E4CFNV3H6VXYXWO72IARYLQ, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.676529733 (528h41m58.547612768s) [noise=true , address=45.91.116.7:4012 , piece_id=L5245B7VAWC4OFEXVUVTAO5QTCWFHBXM6IMZDJZ4KYJQOYPPYGOA, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.724049649 (528h41m58.595132694s) [noise=true , address=77.90.31.10:4519 , piece_id=ZZGKFH6GJS3QZNRXUEIK5CUZ6IQYZ6EDFOJS6WUWPGFEBN5UTX6Q, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.741523653 (528h41m58.612606698s) [noise=true , address=45.91.116.14:4190 , piece_id=CYNBDIE6ICWWXQDGFE4X32ACKUNUZ3XENW7ARAD23UNANRLT5VFA, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.741545114 (528h41m58.612628159s) [seg_pos={0 0}, upload=463709]: Segment reached optimal threshold of 13 pieces. Dec 27 16:10:58.741915188 (528h41m58.612998243s) [noise=true , address=77.90.31.19:4844 , piece_id=ZZIHQCH6BYGR4ICUYEWEMSG7CHEHXZFI5UZ5J4CW2ZK5O4JTJ67Q, seg_pos={0 0}, upload=463709]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:57060->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:57060->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57060->77.90.31.19:4844: use of closed network connection Dec 27 16:10:58.741923425 (528h41m58.61300646s) [noise=true , address=77.90.31.19:4844 , piece_id=ZZIHQCH6BYGR4ICUYEWEMSG7CHEHXZFI5UZ5J4CW2ZK5O4JTJ67Q, seg_pos={0 0}, upload=463709]: Long tail cancellation due to optimal threshold Dec 27 16:10:58.742288279 (528h41m58.613371314s) [noise=true , address=45.91.116.13:4156 , piece_id=LKRWLMDO7WWOIXQNQEZBHSURCURKET7FJ2Q5BPCRKFK6UMSSWOJA, seg_pos={0 0}, upload=463709]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 16:10:58.742296074 (528h41m58.613379109s) [noise=true , address=45.91.116.13:4156 , piece_id=LKRWLMDO7WWOIXQNQEZBHSURCURKET7FJ2Q5BPCRKFK6UMSSWOJA, seg_pos={0 0}, upload=463709]: Long tail cancellation due to optimal threshold Dec 27 16:10:58.742469874 (528h41m58.61355292s) [noise=true , address=77.90.31.15:4697 , piece_id=R4I5W5DSS6R5AQCVAQYSIBAR4OMYSAAHYYZRKHWPA5NL62MRP4NA, seg_pos={0 0}, upload=463709]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 27 16:10:58.742477139 (528h41m58.613560184s) [noise=true , address=77.90.31.15:4697 , piece_id=R4I5W5DSS6R5AQCVAQYSIBAR4OMYSAAHYYZRKHWPA5NL62MRP4NA, seg_pos={0 0}, upload=463709]: Long tail cancellation due to optimal threshold Dec 27 16:10:58.742495514 (528h41m58.613578559s) [seg_pos={0 0}, upload=463709]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:10:58.742569665 (528h41m58.61365271s) [upload=463709]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:10:58.742575325 (528h41m58.61365837s) [upload=463709]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:10:59.032798984 (528h41m58.903882019s) [upload=463709]: Done uploading segments... Dec 27 16:10:59.032833740 (528h41m58.903916785s) [upload=463709]: Upload finished. err: <nil> Duration: 6.380422321s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [....X.X..X..XXX...........X.......X..X.XX..X..X....XX.X....X] duration: 30.004664339s trace: https://jaeger.dev.storj.io/trace/7fd0ee7047f5c08e FAIL (2025-12-27 16:11:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: O42ZCYNIMYDWTOHMWJVOA6FV4SSK2E4CFNV3H6VXYXWO72IARYLQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: IHJZGTHXEFDYNUKE2CMSZKLSIW5LTMDMLUP63CWGNTMTHGHYVP6Q) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: CYNBDIE6ICWWXQDGFE4X32ACKUNUZ3XENW7ARAD23UNANRLT5VFA) context deadline exceeded ======LOGS====== Dec 27 16:10:56.181719477 (528h41m56.052802522s) [upload=463709]: Starting upload Dec 27 16:10:56.181930308 (528h41m56.053013353s) [upload=463709]: Done starting upload Dec 27 16:10:56.182030688 (528h41m56.053113723s) [upload=463709]: Uploading segments... Dec 27 16:10:56.182119939 (528h41m56.053202984s) [upload=463709]: Got next segment. Inline: false Dec 27 16:10:56.182176777 (528h41m56.053259822s) [upload=463709]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:10:56.182183340 (528h41m56.053266385s) [upload=463709]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:10:56.423123263 (528h41m56.294206308s) [seg_pos={0 0}, upload=463709]: Begin upload segment... Dec 27 16:10:56.423180442 (528h41m56.294263477s) [seg_pos={0 0}, upload=463709]: Done begin upload segment. Dec 27 16:10:56.423215959 (528h41m56.294298994s) [noise=true , address=77.90.31.21:4877 , piece_id=O42ZCYNIMYDWTOHMWJVOA6FV4SSK2E4CFNV3H6VXYXWO72IARYLQ, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.423229645 (528h41m56.29431269s) [noise=true , address=45.91.116.7:4012 , piece_id=L5245B7VAWC4OFEXVUVTAO5QTCWFHBXM6IMZDJZ4KYJQOYPPYGOA, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.423670715 (528h41m56.29475375s) [noise=true , address=145.79.227.48:4301 , piece_id=LPSMU4EFWM2LUCUZWYNHVC3Q22DJG2BHZBAXNONQBP6463ZXBM2Q, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.424568335 (528h41m56.29565138s) [noise=true , address=45.91.116.13:4156 , piece_id=LKRWLMDO7WWOIXQNQEZBHSURCURKET7FJ2Q5BPCRKFK6UMSSWOJA, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.424862064 (528h41m56.295945109s) [noise=true , address=77.90.31.19:4844 , piece_id=ZZIHQCH6BYGR4ICUYEWEMSG7CHEHXZFI5UZ5J4CW2ZK5O4JTJ67Q, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.425136466 (528h41m56.296219511s) [noise=true , address=45.91.116.14:4190 , piece_id=CYNBDIE6ICWWXQDGFE4X32ACKUNUZ3XENW7ARAD23UNANRLT5VFA, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.425497003 (528h41m56.296580038s) [noise=true , address=145.79.227.5:4223 , piece_id=CABTYGFXJNPQHTNN2EVYEJQP4Y3ONH3UGHGU2I73U73EHUKR5NIQ, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.425999510 (528h41m56.297082545s) [noise=true , address=145.79.227.37:4566 , piece_id=5FFP6FG6EWG5AGDD22FJ3ZVXU5LK3DCZASZUIMPMDXYNA7HYKUYA, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.426379493 (528h41m56.297462528s) [noise=true , address=145.79.227.16:4912 , piece_id=DLYJQNA3Z7SRLFOI4ZXNVBE5WPRI27TH3CVY4VZSQ2IEXDQJKHNQ, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.427552357 (528h41m56.298635402s) [noise=true , address=145.79.227.36:4519 , piece_id=SPQFN365JO3KCD22GO3JP6QXKXHMRINNIVV4S7WBE4Q5EJQUZWNA, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.428678170 (528h41m56.299761216s) [noise=true , address=145.79.227.51:4411 , piece_id=6SAIOEKXELQAI6ESPRGEAJ46S4KKUCVD3ZUEVBIJ5MWVCWSF6MZA, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.429903864 (528h41m56.300986899s) [noise=true , address=77.90.31.16:4736 , piece_id=IHJZGTHXEFDYNUKE2CMSZKLSIW5LTMDMLUP63CWGNTMTHGHYVP6Q, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.434390167 (528h41m56.305473202s) [noise=true , address=77.90.31.15:4697 , piece_id=R4I5W5DSS6R5AQCVAQYSIBAR4OMYSAAHYYZRKHWPA5NL62MRP4NA, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.438281928 (528h41m56.309364973s) [noise=true , address=145.79.227.15:4876 , piece_id=RYZDEKIZ7J42DLSUN4JMQVITHBAIAOZZLNOD3T75DTBWO7QBYLZQ, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.440522083 (528h41m56.311605118s) [noise=true , address=77.90.31.10:4519 , piece_id=ZZGKFH6GJS3QZNRXUEIK5CUZ6IQYZ6EDFOJS6WUWPGFEBN5UTX6Q, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:56.442575884 (528h41m56.313658929s) [noise=true , address=145.79.227.47:4265 , piece_id=6KKF6M25OJQXKBRP2EJWYASMDNV57RO63NENZ3HXMUEWZPZL4FGA, seg_pos={0 0}, upload=463709]: Uploading piece... Dec 27 16:10:58.059945125 (528h41m57.93102817s) [upload=463709]: Next returned nil segment Dec 27 16:10:58.059964573 (528h41m57.931047638s) [upload=463709]: Waiting for error group managing segments... Dec 27 16:10:58.442778428 (528h41m58.313861473s) [noise=true , address=145.79.227.15:4876 , piece_id=RYZDEKIZ7J42DLSUN4JMQVITHBAIAOZZLNOD3T75DTBWO7QBYLZQ, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.444001778 (528h41m58.315084813s) [noise=true , address=145.79.227.37:4566 , piece_id=5FFP6FG6EWG5AGDD22FJ3ZVXU5LK3DCZASZUIMPMDXYNA7HYKUYA, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.444087872 (528h41m58.315170917s) [noise=true , address=145.79.227.48:4301 , piece_id=LPSMU4EFWM2LUCUZWYNHVC3Q22DJG2BHZBAXNONQBP6463ZXBM2Q, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.444465411 (528h41m58.315548456s) [noise=true , address=145.79.227.5:4223 , piece_id=CABTYGFXJNPQHTNN2EVYEJQP4Y3ONH3UGHGU2I73U73EHUKR5NIQ, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.450753845 (528h41m58.32183688s) [noise=true , address=145.79.227.16:4912 , piece_id=DLYJQNA3Z7SRLFOI4ZXNVBE5WPRI27TH3CVY4VZSQ2IEXDQJKHNQ, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.456006076 (528h41m58.327089121s) [noise=true , address=145.79.227.51:4411 , piece_id=6SAIOEKXELQAI6ESPRGEAJ46S4KKUCVD3ZUEVBIJ5MWVCWSF6MZA, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.456607192 (528h41m58.327690227s) [noise=true , address=145.79.227.36:4519 , piece_id=SPQFN365JO3KCD22GO3JP6QXKXHMRINNIVV4S7WBE4Q5EJQUZWNA, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.477440491 (528h41m58.348523536s) [noise=true , address=145.79.227.47:4265 , piece_id=6KKF6M25OJQXKBRP2EJWYASMDNV57RO63NENZ3HXMUEWZPZL4FGA, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.637293700 (528h41m58.508376745s) [noise=true , address=77.90.31.16:4736 , piece_id=IHJZGTHXEFDYNUKE2CMSZKLSIW5LTMDMLUP63CWGNTMTHGHYVP6Q, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.670869946 (528h41m58.541952981s) [noise=true , address=77.90.31.21:4877 , piece_id=O42ZCYNIMYDWTOHMWJVOA6FV4SSK2E4CFNV3H6VXYXWO72IARYLQ, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.676529733 (528h41m58.547612768s) [noise=true , address=45.91.116.7:4012 , piece_id=L5245B7VAWC4OFEXVUVTAO5QTCWFHBXM6IMZDJZ4KYJQOYPPYGOA, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.724049649 (528h41m58.595132694s) [noise=true , address=77.90.31.10:4519 , piece_id=ZZGKFH6GJS3QZNRXUEIK5CUZ6IQYZ6EDFOJS6WUWPGFEBN5UTX6Q, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.741523653 (528h41m58.612606698s) [noise=true , address=45.91.116.14:4190 , piece_id=CYNBDIE6ICWWXQDGFE4X32ACKUNUZ3XENW7ARAD23UNANRLT5VFA, seg_pos={0 0}, upload=463709]: Done uploading piece. err: <nil> Dec 27 16:10:58.741545114 (528h41m58.612628159s) [seg_pos={0 0}, upload=463709]: Segment reached optimal threshold of 13 pieces. Dec 27 16:10:58.741915188 (528h41m58.612998243s) [noise=true , address=77.90.31.19:4844 , piece_id=ZZIHQCH6BYGR4ICUYEWEMSG7CHEHXZFI5UZ5J4CW2ZK5O4JTJ67Q, seg_pos={0 0}, upload=463709]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: write tcp 188.166.225.170:57060->77.90.31.19:4844: use of closed network connection; write tcp 188.166.225.170:57060->77.90.31.19:4844: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57060->77.90.31.19:4844: use of closed network connection Dec 27 16:10:58.741923425 (528h41m58.61300646s) [noise=true , address=77.90.31.19:4844 , piece_id=ZZIHQCH6BYGR4ICUYEWEMSG7CHEHXZFI5UZ5J4CW2ZK5O4JTJ67Q, seg_pos={0 0}, upload=463709]: Long tail cancellation due to optimal threshold Dec 27 16:10:58.742288279 (528h41m58.613371314s) [noise=true , address=45.91.116.13:4156 , piece_id=LKRWLMDO7WWOIXQNQEZBHSURCURKET7FJ2Q5BPCRKFK6UMSSWOJA, seg_pos={0 0}, upload=463709]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 27 16:10:58.742296074 (528h41m58.613379109s) [noise=true , address=45.91.116.13:4156 , piece_id=LKRWLMDO7WWOIXQNQEZBHSURCURKET7FJ2Q5BPCRKFK6UMSSWOJA, seg_pos={0 0}, upload=463709]: Long tail cancellation due to optimal threshold Dec 27 16:10:58.742469874 (528h41m58.61355292s) [noise=true , address=77.90.31.15:4697 , piece_id=R4I5W5DSS6R5AQCVAQYSIBAR4OMYSAAHYYZRKHWPA5NL62MRP4NA, seg_pos={0 0}, upload=463709]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 27 16:10:58.742477139 (528h41m58.613560184s) [noise=true , address=77.90.31.15:4697 , piece_id=R4I5W5DSS6R5AQCVAQYSIBAR4OMYSAAHYYZRKHWPA5NL62MRP4NA, seg_pos={0 0}, upload=463709]: Long tail cancellation due to optimal threshold Dec 27 16:10:58.742495514 (528h41m58.613578559s) [seg_pos={0 0}, upload=463709]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:10:58.742569665 (528h41m58.61365271s) [upload=463709]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:10:58.742575325 (528h41m58.61365837s) [upload=463709]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:10:59.032798984 (528h41m58.903882019s) [upload=463709]: Done uploading segments... Dec 27 16:10:59.032833740 (528h41m58.903916785s) [upload=463709]: Upload finished. err: <nil> Duration: 30.004687323s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.....XX......X................XX...........................] duration: 52.085545308s trace: https://jaeger.dev.storj.io/trace/9fd07cafe8f934a FAIL (2025-12-27 16:11:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315411726 (528h16m20.186494771s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315905206 (528h16m20.186988251s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316356304 (528h16m20.187439339s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316793537 (528h16m20.187876582s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.317233315 (528h16m20.18831636s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318042255 (528h16m20.1891253s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318150922 (528h16m20.189233967s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318513712 (528h16m20.189596747s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318955464 (528h16m20.190038509s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319037570 (528h16m20.190120605s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319389541 (528h16m20.190472576s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319874033 (528h16m20.190957078s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320378844 (528h16m20.191461879s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320898604 (528h16m20.191981649s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.321434254 (528h16m20.192517289s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:21.620856000 (528h16m21.491939035s) [upload=463344]: Next returned nil segment Dec 27 15:45:21.620875698 (528h16m21.491958743s) [upload=463344]: Waiting for error group managing segments... Dec 27 15:45:21.953178480 (528h16m21.824261516s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.194632602 (528h16m22.065715647s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.195118968 (528h16m22.066202013s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.204573761 (528h16m22.075656806s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.213413343 (528h16m22.084496388s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.372654386 (528h16m22.243737431s) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.473126101 (528h16m22.344209146s) [noise=true , address=77.90.31.12:4615 , piece_id=NTYP2JTUJYSBE4NBMQHTEAY2DA333CR7V3622PVQ3OGRMSPCJK3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.484926481 (528h16m22.356009536s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.563981276 (528h16m22.435064311s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.893470090 (528h16m22.764553126s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.105921755 (528h16m22.9770048s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.153398930 (528h16m23.024481975s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.218328398 (528h16m23.089411443s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244693782 (528h16m23.115776827s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244714231 (528h16m23.115797266s) [seg_pos={0 0}, upload=463344]: Segment reached optimal threshold of 14 pieces. Dec 27 15:45:23.244907037 (528h16m23.115990073s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.244913159 (528h16m23.115996204s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245080929 (528h16m23.116163964s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245087220 (528h16m23.116170275s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245280880 (528h16m23.116363925s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection Dec 27 15:45:23.245370850 (528h16m23.116453885s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245453588 (528h16m23.116536644s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f2MspBPykwYCWAYzJ9PCAXop6mPNarJt3fa6fWZoLipZZQdy1): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245474488 (528h16m23.116557533s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245514535 (528h16m23.11659757s) [seg_pos={0 0}, upload=463344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:45:23.246027922 (528h16m23.117110977s) [upload=463344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:45:23.246044133 (528h16m23.117127168s) [upload=463344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:45:23.560988923 (528h16m23.432071968s) [upload=463344]: Done uploading segments... Dec 27 15:45:23.561009573 (528h16m23.432092618s) [upload=463344]: Upload finished. err: <nil> Duration: 52.085561799s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [.X.....X...XX.XXXX....XXXX..XX..XX......X......XX...X.......] duration: 3.613592309s trace: https://jaeger.dev.storj.io/trace/36696332d68ad959 FAIL (2025-12-27 16:11:00 UTC) orbiter warning: context deadline exceeded Duration: 3.613617236s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [..X.....X...XX.XXXX....XXXX..XX..XX......X......XX...X......] duration: 3.598984792s trace: https://jaeger.dev.storj.io/trace/36696332d68ad959 FAIL (2025-12-27 16:10:30 UTC) orbiter warning: context deadline exceeded Duration: 3.59900947s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X.....XX......X................XX..........................] duration: 55.45374475s trace: https://jaeger.dev.storj.io/trace/9fd07cafe8f934a FAIL (2025-12-27 16:10:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315411726 (528h16m20.186494771s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315905206 (528h16m20.186988251s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316356304 (528h16m20.187439339s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316793537 (528h16m20.187876582s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.317233315 (528h16m20.18831636s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318042255 (528h16m20.1891253s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318150922 (528h16m20.189233967s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318513712 (528h16m20.189596747s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318955464 (528h16m20.190038509s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319037570 (528h16m20.190120605s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319389541 (528h16m20.190472576s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319874033 (528h16m20.190957078s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320378844 (528h16m20.191461879s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320898604 (528h16m20.191981649s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.321434254 (528h16m20.192517289s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:21.620856000 (528h16m21.491939035s) [upload=463344]: Next returned nil segment Dec 27 15:45:21.620875698 (528h16m21.491958743s) [upload=463344]: Waiting for error group managing segments... Dec 27 15:45:21.953178480 (528h16m21.824261516s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.194632602 (528h16m22.065715647s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.195118968 (528h16m22.066202013s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.204573761 (528h16m22.075656806s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.213413343 (528h16m22.084496388s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.372654386 (528h16m22.243737431s) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.473126101 (528h16m22.344209146s) [noise=true , address=77.90.31.12:4615 , piece_id=NTYP2JTUJYSBE4NBMQHTEAY2DA333CR7V3622PVQ3OGRMSPCJK3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.484926481 (528h16m22.356009536s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.563981276 (528h16m22.435064311s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.893470090 (528h16m22.764553126s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.105921755 (528h16m22.9770048s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.153398930 (528h16m23.024481975s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.218328398 (528h16m23.089411443s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244693782 (528h16m23.115776827s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244714231 (528h16m23.115797266s) [seg_pos={0 0}, upload=463344]: Segment reached optimal threshold of 14 pieces. Dec 27 15:45:23.244907037 (528h16m23.115990073s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.244913159 (528h16m23.115996204s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245080929 (528h16m23.116163964s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245087220 (528h16m23.116170275s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245280880 (528h16m23.116363925s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection Dec 27 15:45:23.245370850 (528h16m23.116453885s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245453588 (528h16m23.116536644s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f2MspBPykwYCWAYzJ9PCAXop6mPNarJt3fa6fWZoLipZZQdy1): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245474488 (528h16m23.116557533s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245514535 (528h16m23.11659757s) [seg_pos={0 0}, upload=463344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:45:23.246027922 (528h16m23.117110977s) [upload=463344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:45:23.246044133 (528h16m23.117127168s) [upload=463344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:45:23.560988923 (528h16m23.432071968s) [upload=463344]: Done uploading segments... Dec 27 15:45:23.561009573 (528h16m23.432092618s) [upload=463344]: Upload finished. err: <nil> Duration: 55.453759578s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.....X.X..X..XXX...........X.......X..X.XX..X..X....XX.X....] duration: 5.776989734s trace: https://jaeger.dev.storj.io/trace/623a6571e7aec859 FAIL (2025-12-27 16:10:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: VCIT67WD6N7WBALGTQ4RY7NGBZE2RHLNHNXTCLYVVOQQSK44U54A) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: WQZTMUIA25SXDRS77VPYTSCRWARWLXZBSW4AD7QSJ6HA4OBUJ3HA) context deadline exceeded ======LOGS====== Dec 27 16:06:35.474921951 (528h37m35.346004996s) [upload=463652]: Starting upload Dec 27 16:06:35.474981193 (528h37m35.346064228s) [upload=463652]: Done starting upload Dec 27 16:06:35.475115549 (528h37m35.346198584s) [upload=463652]: Uploading segments... Dec 27 16:06:35.475258982 (528h37m35.346342027s) [upload=463652]: Got next segment. Inline: false Dec 27 16:06:35.475335589 (528h37m35.346418634s) [upload=463652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:06:35.475342923 (528h37m35.346425968s) [upload=463652]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:06:35.726931893 (528h37m35.598014958s) [seg_pos={0 0}, upload=463652]: Begin upload segment... Dec 27 16:06:35.727004631 (528h37m35.598087666s) [seg_pos={0 0}, upload=463652]: Done begin upload segment. Dec 27 16:06:35.727059677 (528h37m35.598142722s) [noise=true , address=145.79.227.11:4732 , piece_id=JQKZYXIF53JUCBUECYHUOHBE7MRVSCVTBU257S2SGSOBRAEJZHJA, seg_pos={0 0}, upload=463652]: Uploading piece... Dec 27 16:06:35.727071489 (528h37m35.598154524s) [noise=true , address=45.91.116.8:4045 , piece_id=IQSF3AF6PRKBMOI7BDOIWMZCIDPL3IXQ24OLRJMBKMBVHMQXN3OQ, seg_pos={0 0}, upload=463652]: Uploading piece... Dec 27 16:06:35.727139258 (528h37m35.598222303s) [noise=true , address=145.79.227.35:4482 , piece_id=VPEX4G6IKH4TTJYR5DVD5IXR54WXF7EZK6BZ6VOMW6UL3PJDT7JA, seg_pos={0 0}, upload=463652]: Uploading piece... Dec 27 16:06:35.727943219 (528h37m35.599026274s) [noise=true , address=45.91.116.6:4974 , piece_id=VCIT67WD6N7WBALGTQ4RY7NGBZE2RHLNHNXTCLYVVOQQSK44U54A, seg_pos={0 0}, upload=463652]: Uploading piece... Dec 27 16:06:35.728400149 (528h37m35.599483184s) [noise=true , address=45.91.116.13:4155 , piece_id=6P7MZBZG434FH62MVYGSFCWSMVWLTGH2ZTJ6TXCPZF26D3TBGIEQ, seg_pos={0 0}, upload=463652]: Uploading piece... Dec 27 16:06:35.728830829 (528h37m35.599913864s) [noise=true , address=77.90.31.16:4736 , piece_id=WQZTMUIA25SXDRS77VPYTSCRWARWLXZBSW4AD7QSJ6HA4OBUJ3HA, seg_pos={0 0}, upload=463652]: Uploading piece... Dec 27 16:06:35.729262100 (528h37m35.600345135s) [noise=true , address=77.90.31.13:4627 , piece_id=6M6ZMZDEPKAUVUUNLURYCLI5Q4MAV6IDHU3JTALGUQVYRDR5KODQ, seg_pos={0 0}, upload=463652]: Uploading piece... Dec 27 16:06:35.729856282 (528h37m35.600939317s) [noise=true , address=145.79.227.13:4802 , piece_id=RYRCUVNGBBFNWKCXAZ6YEQT4D5OOBBJ4ODMZKNUWP4QNNWSSAZQA, seg_pos={0 0}, upload=463652]: Uploading piece... Dec 27 16:06:35.730221127 (528h37m35.601304162s) [noise=true , address=145.79.227.41:4085 , piece_id=EUOI5Z3IV7PF5LIQWBDJSIFUTAJWWLHMHTNEBIK7M4OTINQ2TEPA, seg_pos={0 0}, upload=463652]: Uploading piece... Dec 27 16:06:35.730619675 (528h37m35.60170271s) [noise=true , address=145.79.227.8:4329 , piece_id=6RIAO23XJBDPGM65PCRCDUWW2UXCKXQFHHC6BUFBCKWBKDDVWLRA, seg_pos={0 0}, upload=463652]: Uploading piece... Dec 27 16:06:35.732407209 (528h37m35.603490244s) [noise=true , address=145.79.227.24:4581 , piece_id=AH7EAOY6GKIWTGQBCG22QKELTWOKB2WTAXOGQ7FRQHTUT2I3KLLQ, seg_pos={0 0}, upload=463652]: Uploading piece... Dec 27 16:06:35.735801412 (528h37m35.606884447s) [noise=true , address=145.79.227.34:4906 , piece_id=663EPKOLUDELCBCC7QCQF6WZKLGCOORV7MWK53RSR63KSKQTUBJA, seg_pos={0 0}, upload=463652]: Uploading piece... Dec 27 16:06:35.738484491 (528h37m35.609567526s) [noise=true , address=45.91.116.9:4263 , piece_id=HXT4T4XTESEIPI5OWX65TPSJUNJ53BCDVOVSOK3ZRTUMI362YT3Q, seg_pos={0 0}, upload=463652]: Uploading piece... Dec 27 16:06:35.740616090 (528h37m35.611699135s) [noise=true , address=145.79.227.52:4447 , piece_id=A3Y7HB6C3QYMA3FHMBQ4A2I3ACYAUX65VOA2NKEF2L2DOYDBH3AQ, seg_pos={0 0}, upload=463652]: Uploading piece... Dec 27 16:06:35.745199688 (528h37m35.616282733s) [noise=true , address=145.79.227.39:4015 , piece_id=EYWK4KMFNUEMM6HBJULZZ6GRBY4KOCCWEBPIDMQ726KQBL6X27TQ, seg_pos={0 0}, upload=463652]: Uploading piece... Dec 27 16:06:35.746184914 (528h37m35.617267949s) [noise=true , address=45.91.116.5:4300 , piece_id=RKOPTM6XEAQKM7RQRROTKF2P53LOMCBLXEIPCKRCYQYXEM74WQAQ, seg_pos={0 0}, upload=463652]: Uploading piece... Dec 27 16:06:37.417553830 (528h37m37.288636865s) [upload=463652]: Next returned nil segment Dec 27 16:06:37.417588306 (528h37m37.288671351s) [upload=463652]: Waiting for error group managing segments... Dec 27 16:06:37.723308853 (528h37m37.594391888s) [noise=true , address=145.79.227.13:4802 , piece_id=RYRCUVNGBBFNWKCXAZ6YEQT4D5OOBBJ4ODMZKNUWP4QNNWSSAZQA, seg_pos={0 0}, upload=463652]: Done uploading piece. err: <nil> Dec 27 16:06:37.726701162 (528h37m37.597784197s) [noise=true , address=145.79.227.8:4329 , piece_id=6RIAO23XJBDPGM65PCRCDUWW2UXCKXQFHHC6BUFBCKWBKDDVWLRA, seg_pos={0 0}, upload=463652]: Done uploading piece. err: <nil> Dec 27 16:06:37.733191050 (528h37m37.604274085s) [noise=true , address=145.79.227.11:4732 , piece_id=JQKZYXIF53JUCBUECYHUOHBE7MRVSCVTBU257S2SGSOBRAEJZHJA, seg_pos={0 0}, upload=463652]: Done uploading piece. err: <nil> Dec 27 16:06:37.739062690 (528h37m37.610145735s) [noise=true , address=145.79.227.41:4085 , piece_id=EUOI5Z3IV7PF5LIQWBDJSIFUTAJWWLHMHTNEBIK7M4OTINQ2TEPA, seg_pos={0 0}, upload=463652]: Done uploading piece. err: <nil> Dec 27 16:06:37.743573210 (528h37m37.614656255s) [noise=true , address=145.79.227.52:4447 , piece_id=A3Y7HB6C3QYMA3FHMBQ4A2I3ACYAUX65VOA2NKEF2L2DOYDBH3AQ, seg_pos={0 0}, upload=463652]: Done uploading piece. err: <nil> Dec 27 16:06:37.750970976 (528h37m37.622054041s) [noise=true , address=145.79.227.39:4015 , piece_id=EYWK4KMFNUEMM6HBJULZZ6GRBY4KOCCWEBPIDMQ726KQBL6X27TQ, seg_pos={0 0}, upload=463652]: Done uploading piece. err: <nil> Dec 27 16:06:37.758816075 (528h37m37.62989911s) [noise=true , address=145.79.227.24:4581 , piece_id=AH7EAOY6GKIWTGQBCG22QKELTWOKB2WTAXOGQ7FRQHTUT2I3KLLQ, seg_pos={0 0}, upload=463652]: Done uploading piece. err: <nil> Dec 27 16:06:37.775466610 (528h37m37.646549655s) [noise=true , address=145.79.227.35:4482 , piece_id=VPEX4G6IKH4TTJYR5DVD5IXR54WXF7EZK6BZ6VOMW6UL3PJDT7JA, seg_pos={0 0}, upload=463652]: Done uploading piece. err: <nil> Dec 27 16:06:37.913232111 (528h37m37.784315156s) [noise=true , address=77.90.31.16:4736 , piece_id=WQZTMUIA25SXDRS77VPYTSCRWARWLXZBSW4AD7QSJ6HA4OBUJ3HA, seg_pos={0 0}, upload=463652]: Done uploading piece. err: <nil> Dec 27 16:06:37.998701690 (528h37m37.869784745s) [noise=true , address=77.90.31.13:4627 , piece_id=6M6ZMZDEPKAUVUUNLURYCLI5Q4MAV6IDHU3JTALGUQVYRDR5KODQ, seg_pos={0 0}, upload=463652]: Done uploading piece. err: <nil> Dec 27 16:06:38.011484992 (528h37m37.882568037s) [noise=true , address=45.91.116.6:4974 , piece_id=VCIT67WD6N7WBALGTQ4RY7NGBZE2RHLNHNXTCLYVVOQQSK44U54A, seg_pos={0 0}, upload=463652]: Done uploading piece. err: <nil> Dec 27 16:06:38.021131119 (528h37m37.892214154s) [noise=true , address=45.91.116.8:4045 , piece_id=IQSF3AF6PRKBMOI7BDOIWMZCIDPL3IXQ24OLRJMBKMBVHMQXN3OQ, seg_pos={0 0}, upload=463652]: Done uploading piece. err: <nil> Dec 27 16:06:38.048845793 (528h37m37.919928838s) [noise=true , address=45.91.116.5:4300 , piece_id=RKOPTM6XEAQKM7RQRROTKF2P53LOMCBLXEIPCKRCYQYXEM74WQAQ, seg_pos={0 0}, upload=463652]: Done uploading piece. err: <nil> Dec 27 16:06:38.048869438 (528h37m37.919952483s) [seg_pos={0 0}, upload=463652]: Segment reached optimal threshold of 13 pieces. Dec 27 16:06:38.049118923 (528h37m37.920201968s) [noise=true , address=145.79.227.34:4906 , piece_id=663EPKOLUDELCBCC7QCQF6WZKLGCOORV7MWK53RSR63KSKQTUBJA, seg_pos={0 0}, upload=463652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 27 16:06:38.049130736 (528h37m37.920213781s) [noise=true , address=145.79.227.34:4906 , piece_id=663EPKOLUDELCBCC7QCQF6WZKLGCOORV7MWK53RSR63KSKQTUBJA, seg_pos={0 0}, upload=463652]: Long tail cancellation due to optimal threshold Dec 27 16:06:38.049211329 (528h37m37.920294364s) [noise=true , address=45.91.116.9:4263 , piece_id=HXT4T4XTESEIPI5OWX65TPSJUNJ53BCDVOVSOK3ZRTUMI362YT3Q, seg_pos={0 0}, upload=463652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 27 16:06:38.049221599 (528h37m37.920304654s) [noise=true , address=45.91.116.9:4263 , piece_id=HXT4T4XTESEIPI5OWX65TPSJUNJ53BCDVOVSOK3ZRTUMI362YT3Q, seg_pos={0 0}, upload=463652]: Long tail cancellation due to optimal threshold Dec 27 16:06:38.049403264 (528h37m37.920486309s) [noise=true , address=45.91.116.13:4155 , piece_id=6P7MZBZG434FH62MVYGSFCWSMVWLTGH2ZTJ6TXCPZF26D3TBGIEQ, seg_pos={0 0}, upload=463652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 27 16:06:38.049413585 (528h37m37.92049663s) [noise=true , address=45.91.116.13:4155 , piece_id=6P7MZBZG434FH62MVYGSFCWSMVWLTGH2ZTJ6TXCPZF26D3TBGIEQ, seg_pos={0 0}, upload=463652]: Long tail cancellation due to optimal threshold Dec 27 16:06:38.049488848 (528h37m37.920571893s) [seg_pos={0 0}, upload=463652]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:06:38.049576164 (528h37m37.920659199s) [upload=463652]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:06:38.049584210 (528h37m37.920667255s) [upload=463652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:06:38.344663683 (528h37m38.215746758s) [upload=463652]: Done uploading segments... Dec 27 16:06:38.344693761 (528h37m38.215776806s) [upload=463652]: Upload finished. err: <nil> Duration: 5.777018098s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [......X.X..X..XXX...........X.......X..X.XX..X..X....XX.X...] duration: 5.718555946s trace: https://jaeger.dev.storj.io/trace/623a6571e7aec859 FAIL (2025-12-27 16:09:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, Piece ID: VCIT67WD6N7WBALGTQ4RY7NGBZE2RHLNHNXTCLYVVOQQSK44U54A) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: WQZTMUIA25SXDRS77VPYTSCRWARWLXZBSW4AD7QSJ6HA4OBUJ3HA) context deadline exceeded ======LOGS====== Dec 27 16:06:35.474921951 (528h37m35.346004996s) [upload=463652]: Starting upload Dec 27 16:06:35.474981193 (528h37m35.346064228s) [upload=463652]: Done starting upload Dec 27 16:06:35.475115549 (528h37m35.346198584s) [upload=463652]: Uploading segments... Dec 27 16:06:35.475258982 (528h37m35.346342027s) [upload=463652]: Got next segment. Inline: false Dec 27 16:06:35.475335589 (528h37m35.346418634s) [upload=463652]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:06:35.475342923 (528h37m35.346425968s) [upload=463652]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:06:35.726931893 (528h37m35.598014958s) [seg_pos={0 0}, upload=463652]: Begin upload segment... Dec 27 16:06:35.727004631 (528h37m35.598087666s) [seg_pos={0 0}, upload=463652]: Done begin upload segment. Dec 27 16:06:35.727059677 (528h37m35.598142722s) [noise=true , address=145.79.227.11:4732 , piece_id=JQKZYXIF53JUCBUECYHUOHBE7MRVSCVTBU257S2SGSOBRAEJZHJA, seg_pos={0 0}, upload=463652]: Uploading piece... Dec 27 16:06:35.727071489 (528h37m35.598154524s) [noise=true , address=45.91.116.8:4045 , piece_id=IQSF3AF6PRKBMOI7BDOIWMZCIDPL3IXQ24OLRJMBKMBVHMQXN3OQ, seg_pos={0 0}, upload=463652]: Uploading piece... Dec 27 16:06:35.727139258 (528h37m35.598222303s) [noise=true , address=145.79.227.35:4482 , piece_id=VPEX4G6IKH4TTJYR5DVD5IXR54WXF7EZK6BZ6VOMW6UL3PJDT7JA, seg_pos={0 0}, upload=463652]: Uploading piece... Dec 27 16:06:35.727943219 (528h37m35.599026274s) [noise=true , address=45.91.116.6:4974 , piece_id=VCIT67WD6N7WBALGTQ4RY7NGBZE2RHLNHNXTCLYVVOQQSK44U54A, seg_pos={0 0}, upload=463652]: Uploading piece... Dec 27 16:06:35.728400149 (528h37m35.599483184s) [noise=true , address=45.91.116.13:4155 , piece_id=6P7MZBZG434FH62MVYGSFCWSMVWLTGH2ZTJ6TXCPZF26D3TBGIEQ, seg_pos={0 0}, upload=463652]: Uploading piece... Dec 27 16:06:35.728830829 (528h37m35.599913864s) [noise=true , address=77.90.31.16:4736 , piece_id=WQZTMUIA25SXDRS77VPYTSCRWARWLXZBSW4AD7QSJ6HA4OBUJ3HA, seg_pos={0 0}, upload=463652]: Uploading piece... Dec 27 16:06:35.729262100 (528h37m35.600345135s) [noise=true , address=77.90.31.13:4627 , piece_id=6M6ZMZDEPKAUVUUNLURYCLI5Q4MAV6IDHU3JTALGUQVYRDR5KODQ, seg_pos={0 0}, upload=463652]: Uploading piece... Dec 27 16:06:35.729856282 (528h37m35.600939317s) [noise=true , address=145.79.227.13:4802 , piece_id=RYRCUVNGBBFNWKCXAZ6YEQT4D5OOBBJ4ODMZKNUWP4QNNWSSAZQA, seg_pos={0 0}, upload=463652]: Uploading piece... Dec 27 16:06:35.730221127 (528h37m35.601304162s) [noise=true , address=145.79.227.41:4085 , piece_id=EUOI5Z3IV7PF5LIQWBDJSIFUTAJWWLHMHTNEBIK7M4OTINQ2TEPA, seg_pos={0 0}, upload=463652]: Uploading piece... Dec 27 16:06:35.730619675 (528h37m35.60170271s) [noise=true , address=145.79.227.8:4329 , piece_id=6RIAO23XJBDPGM65PCRCDUWW2UXCKXQFHHC6BUFBCKWBKDDVWLRA, seg_pos={0 0}, upload=463652]: Uploading piece... Dec 27 16:06:35.732407209 (528h37m35.603490244s) [noise=true , address=145.79.227.24:4581 , piece_id=AH7EAOY6GKIWTGQBCG22QKELTWOKB2WTAXOGQ7FRQHTUT2I3KLLQ, seg_pos={0 0}, upload=463652]: Uploading piece... Dec 27 16:06:35.735801412 (528h37m35.606884447s) [noise=true , address=145.79.227.34:4906 , piece_id=663EPKOLUDELCBCC7QCQF6WZKLGCOORV7MWK53RSR63KSKQTUBJA, seg_pos={0 0}, upload=463652]: Uploading piece... Dec 27 16:06:35.738484491 (528h37m35.609567526s) [noise=true , address=45.91.116.9:4263 , piece_id=HXT4T4XTESEIPI5OWX65TPSJUNJ53BCDVOVSOK3ZRTUMI362YT3Q, seg_pos={0 0}, upload=463652]: Uploading piece... Dec 27 16:06:35.740616090 (528h37m35.611699135s) [noise=true , address=145.79.227.52:4447 , piece_id=A3Y7HB6C3QYMA3FHMBQ4A2I3ACYAUX65VOA2NKEF2L2DOYDBH3AQ, seg_pos={0 0}, upload=463652]: Uploading piece... Dec 27 16:06:35.745199688 (528h37m35.616282733s) [noise=true , address=145.79.227.39:4015 , piece_id=EYWK4KMFNUEMM6HBJULZZ6GRBY4KOCCWEBPIDMQ726KQBL6X27TQ, seg_pos={0 0}, upload=463652]: Uploading piece... Dec 27 16:06:35.746184914 (528h37m35.617267949s) [noise=true , address=45.91.116.5:4300 , piece_id=RKOPTM6XEAQKM7RQRROTKF2P53LOMCBLXEIPCKRCYQYXEM74WQAQ, seg_pos={0 0}, upload=463652]: Uploading piece... Dec 27 16:06:37.417553830 (528h37m37.288636865s) [upload=463652]: Next returned nil segment Dec 27 16:06:37.417588306 (528h37m37.288671351s) [upload=463652]: Waiting for error group managing segments... Dec 27 16:06:37.723308853 (528h37m37.594391888s) [noise=true , address=145.79.227.13:4802 , piece_id=RYRCUVNGBBFNWKCXAZ6YEQT4D5OOBBJ4ODMZKNUWP4QNNWSSAZQA, seg_pos={0 0}, upload=463652]: Done uploading piece. err: <nil> Dec 27 16:06:37.726701162 (528h37m37.597784197s) [noise=true , address=145.79.227.8:4329 , piece_id=6RIAO23XJBDPGM65PCRCDUWW2UXCKXQFHHC6BUFBCKWBKDDVWLRA, seg_pos={0 0}, upload=463652]: Done uploading piece. err: <nil> Dec 27 16:06:37.733191050 (528h37m37.604274085s) [noise=true , address=145.79.227.11:4732 , piece_id=JQKZYXIF53JUCBUECYHUOHBE7MRVSCVTBU257S2SGSOBRAEJZHJA, seg_pos={0 0}, upload=463652]: Done uploading piece. err: <nil> Dec 27 16:06:37.739062690 (528h37m37.610145735s) [noise=true , address=145.79.227.41:4085 , piece_id=EUOI5Z3IV7PF5LIQWBDJSIFUTAJWWLHMHTNEBIK7M4OTINQ2TEPA, seg_pos={0 0}, upload=463652]: Done uploading piece. err: <nil> Dec 27 16:06:37.743573210 (528h37m37.614656255s) [noise=true , address=145.79.227.52:4447 , piece_id=A3Y7HB6C3QYMA3FHMBQ4A2I3ACYAUX65VOA2NKEF2L2DOYDBH3AQ, seg_pos={0 0}, upload=463652]: Done uploading piece. err: <nil> Dec 27 16:06:37.750970976 (528h37m37.622054041s) [noise=true , address=145.79.227.39:4015 , piece_id=EYWK4KMFNUEMM6HBJULZZ6GRBY4KOCCWEBPIDMQ726KQBL6X27TQ, seg_pos={0 0}, upload=463652]: Done uploading piece. err: <nil> Dec 27 16:06:37.758816075 (528h37m37.62989911s) [noise=true , address=145.79.227.24:4581 , piece_id=AH7EAOY6GKIWTGQBCG22QKELTWOKB2WTAXOGQ7FRQHTUT2I3KLLQ, seg_pos={0 0}, upload=463652]: Done uploading piece. err: <nil> Dec 27 16:06:37.775466610 (528h37m37.646549655s) [noise=true , address=145.79.227.35:4482 , piece_id=VPEX4G6IKH4TTJYR5DVD5IXR54WXF7EZK6BZ6VOMW6UL3PJDT7JA, seg_pos={0 0}, upload=463652]: Done uploading piece. err: <nil> Dec 27 16:06:37.913232111 (528h37m37.784315156s) [noise=true , address=77.90.31.16:4736 , piece_id=WQZTMUIA25SXDRS77VPYTSCRWARWLXZBSW4AD7QSJ6HA4OBUJ3HA, seg_pos={0 0}, upload=463652]: Done uploading piece. err: <nil> Dec 27 16:06:37.998701690 (528h37m37.869784745s) [noise=true , address=77.90.31.13:4627 , piece_id=6M6ZMZDEPKAUVUUNLURYCLI5Q4MAV6IDHU3JTALGUQVYRDR5KODQ, seg_pos={0 0}, upload=463652]: Done uploading piece. err: <nil> Dec 27 16:06:38.011484992 (528h37m37.882568037s) [noise=true , address=45.91.116.6:4974 , piece_id=VCIT67WD6N7WBALGTQ4RY7NGBZE2RHLNHNXTCLYVVOQQSK44U54A, seg_pos={0 0}, upload=463652]: Done uploading piece. err: <nil> Dec 27 16:06:38.021131119 (528h37m37.892214154s) [noise=true , address=45.91.116.8:4045 , piece_id=IQSF3AF6PRKBMOI7BDOIWMZCIDPL3IXQ24OLRJMBKMBVHMQXN3OQ, seg_pos={0 0}, upload=463652]: Done uploading piece. err: <nil> Dec 27 16:06:38.048845793 (528h37m37.919928838s) [noise=true , address=45.91.116.5:4300 , piece_id=RKOPTM6XEAQKM7RQRROTKF2P53LOMCBLXEIPCKRCYQYXEM74WQAQ, seg_pos={0 0}, upload=463652]: Done uploading piece. err: <nil> Dec 27 16:06:38.048869438 (528h37m37.919952483s) [seg_pos={0 0}, upload=463652]: Segment reached optimal threshold of 13 pieces. Dec 27 16:06:38.049118923 (528h37m37.920201968s) [noise=true , address=145.79.227.34:4906 , piece_id=663EPKOLUDELCBCC7QCQF6WZKLGCOORV7MWK53RSR63KSKQTUBJA, seg_pos={0 0}, upload=463652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12m9gfwSNSLJB2XyBAQUVtWUpc1jQtr5dvYWzHqXeaLQUzyW9cw): protocol: expected piece hash; context canceled; EOF Dec 27 16:06:38.049130736 (528h37m37.920213781s) [noise=true , address=145.79.227.34:4906 , piece_id=663EPKOLUDELCBCC7QCQF6WZKLGCOORV7MWK53RSR63KSKQTUBJA, seg_pos={0 0}, upload=463652]: Long tail cancellation due to optimal threshold Dec 27 16:06:38.049211329 (528h37m37.920294364s) [noise=true , address=45.91.116.9:4263 , piece_id=HXT4T4XTESEIPI5OWX65TPSJUNJ53BCDVOVSOK3ZRTUMI362YT3Q, seg_pos={0 0}, upload=463652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 27 16:06:38.049221599 (528h37m37.920304654s) [noise=true , address=45.91.116.9:4263 , piece_id=HXT4T4XTESEIPI5OWX65TPSJUNJ53BCDVOVSOK3ZRTUMI362YT3Q, seg_pos={0 0}, upload=463652]: Long tail cancellation due to optimal threshold Dec 27 16:06:38.049403264 (528h37m37.920486309s) [noise=true , address=45.91.116.13:4155 , piece_id=6P7MZBZG434FH62MVYGSFCWSMVWLTGH2ZTJ6TXCPZF26D3TBGIEQ, seg_pos={0 0}, upload=463652]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 27 16:06:38.049413585 (528h37m37.92049663s) [noise=true , address=45.91.116.13:4155 , piece_id=6P7MZBZG434FH62MVYGSFCWSMVWLTGH2ZTJ6TXCPZF26D3TBGIEQ, seg_pos={0 0}, upload=463652]: Long tail cancellation due to optimal threshold Dec 27 16:06:38.049488848 (528h37m37.920571893s) [seg_pos={0 0}, upload=463652]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:06:38.049576164 (528h37m37.920659199s) [upload=463652]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:06:38.049584210 (528h37m37.920667255s) [upload=463652]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:06:38.344663683 (528h37m38.215746758s) [upload=463652]: Done uploading segments... Dec 27 16:06:38.344693761 (528h37m38.215776806s) [upload=463652]: Upload finished. err: <nil> Duration: 5.718586965s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X.....XX......X................XX.........................] duration: 50.65535326s trace: https://jaeger.dev.storj.io/trace/9fd07cafe8f934a FAIL (2025-12-27 16:09:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315411726 (528h16m20.186494771s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.315905206 (528h16m20.186988251s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316356304 (528h16m20.187439339s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.316793537 (528h16m20.187876582s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.317233315 (528h16m20.18831636s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318042255 (528h16m20.1891253s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318150922 (528h16m20.189233967s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318513712 (528h16m20.189596747s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.318955464 (528h16m20.190038509s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319037570 (528h16m20.190120605s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319389541 (528h16m20.190472576s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.319874033 (528h16m20.190957078s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320378844 (528h16m20.191461879s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.320898604 (528h16m20.191981649s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:20.321434254 (528h16m20.192517289s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Uploading piece... Dec 27 15:45:21.620856000 (528h16m21.491939035s) [upload=463344]: Next returned nil segment Dec 27 15:45:21.620875698 (528h16m21.491958743s) [upload=463344]: Waiting for error group managing segments... Dec 27 15:45:21.953178480 (528h16m21.824261516s) [noise=true , address=193.149.164.143:4699 , piece_id=IJ47JEICL33JFDTYMQKRVWW4IWJWQWH3Z3JT7U74S23R4REEX3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.194632602 (528h16m22.065715647s) [noise=true , address=145.79.227.46:4633 , piece_id=DOG6JX2LL2FUAQU4EMWRSDE7PKXRZ22HLMSH7RU5FJQVIZUPUI3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.195118968 (528h16m22.066202013s) [noise=true , address=64.49.13.140:4507 , piece_id=OTTOZ76VKDZVCC4FIW2TT5DMFW5DGB6BSZGBR74H6SGPNOYBXOMA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.204573761 (528h16m22.075656806s) [noise=true , address=145.79.227.13:4816 , piece_id=CTWUKKJ3KSREF6R5Y2JWPBSB75RVLG35XFSLFCRQG6MV6DHUJ3LQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.213413343 (528h16m22.084496388s) [noise=true , address=145.79.227.9:4373 , piece_id=IU655TIAJA2GNNGBXDRHCHNG5PCWCWMHOEMSCJAXZGTH2M6GUMPQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.372654386 (528h16m22.243737431s) [noise=true , address=77.90.31.18:4816 , piece_id=IQDNTB546AMC6A6W5ACE5AGGX34VW64HDYF7HOW25PPJ7QHIE5QA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.473126101 (528h16m22.344209146s) [noise=true , address=77.90.31.12:4615 , piece_id=NTYP2JTUJYSBE4NBMQHTEAY2DA333CR7V3622PVQ3OGRMSPCJK3A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.484926481 (528h16m22.356009536s) [noise=true , address=77.90.31.11:4583 , piece_id=YW25ILTYXXYI5GYW4RJQM6JJOYHWC63STBIQSN3SAFI77A2MYCKA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.563981276 (528h16m22.435064311s) [noise=true , address=77.90.31.10:4548 , piece_id=BJVQC6RP22A3FUJIVVKYGSYDCGV2ZTHQ4LW4O7DI4SHCPAW7POWQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:22.893470090 (528h16m22.764553126s) [noise=true , address=77.90.31.14:4671 , piece_id=7QMU4PDJDX5SJ3O5H73TWML7EN5J745XRBODYED3GRVFUEVVHMYQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.105921755 (528h16m22.9770048s) [noise=true , address=145.79.227.44:4196 , piece_id=ZJQUJWVJDKKD4G4ZFPLGCCYEP6SYMIDLRAEHT27XTZYWUV2UI44A, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.153398930 (528h16m23.024481975s) [noise=true , address=145.79.227.34:4933 , piece_id=KJYZXHLUXZOJZYA3KAWUGUSOT5DOSZRSDUZ56VV7TBPNEIXOVOUA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.218328398 (528h16m23.089411443s) [noise=true , address=145.79.227.30:4778 , piece_id=BQ3XFP7FD7BM2E2HHBBOBGN3G2DOTGGH2PJZVQYMD5KTZVXILTQQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244693782 (528h16m23.115776827s) [noise=true , address=145.79.227.40:4069 , piece_id=SYJ3HHMN76A4RHRQK3T3DBMJQH5QVW4RVHKTD7IGF5XBFU227QNA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: <nil> Dec 27 15:45:23.244714231 (528h16m23.115797266s) [seg_pos={0 0}, upload=463344]: Segment reached optimal threshold of 14 pieces. Dec 27 15:45:23.244907037 (528h16m23.115990073s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12a96r9qNJPvk95jgSAibJEaCDDisk6AFGHEC1XeXfhmQewg4ak): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.244913159 (528h16m23.115996204s) [noise=true , address=64.49.13.170:4717 , piece_id=YIKVCMHNJEFWQECCD5TI66WRTU4F5UFD7PL4ELASS3Q2AFP5RNIQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245080929 (528h16m23.116163964s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hkgeCi3efNQFv37chVFEi4v2AyWFEwv9keyy5oz6RkJ4Ubunp): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245087220 (528h16m23.116170275s) [noise=true , address=77.90.31.13:4635 , piece_id=DLV73HNC7GYRB5BLJNM2YWSU4AVN7IV3BOO3PSMCNB4AXHM27YWA, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245280880 (528h16m23.116363925s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52362->64.49.13.169:4706: use of closed network connection Dec 27 15:45:23.245370850 (528h16m23.116453885s) [noise=true , address=64.49.13.169:4706 , piece_id=2VPCLZ5XY4BROYIKDVI2GVCPMS2JDAI5YT3F7FR3L3GAMJ5NJWEQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245453588 (528h16m23.116536644s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1f2MspBPykwYCWAYzJ9PCAXop6mPNarJt3fa6fWZoLipZZQdy1): protocol: expected piece hash; context canceled; EOF Dec 27 15:45:23.245474488 (528h16m23.116557533s) [noise=true , address=45.91.116.14:4223 , piece_id=FVDKKWMFP67BLM3W4NAMPSDYOF7L7YJBQF3R66RUQSGBW4IEDDFQ, seg_pos={0 0}, upload=463344]: Long tail cancellation due to optimal threshold Dec 27 15:45:23.245514535 (528h16m23.11659757s) [seg_pos={0 0}, upload=463344]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 27 15:45:23.246027922 (528h16m23.117110977s) [upload=463344]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 15:45:23.246044133 (528h16m23.117127168s) [upload=463344]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 15:45:23.560988923 (528h16m23.432071968s) [upload=463344]: Done uploading segments... Dec 27 15:45:23.561009573 (528h16m23.432092618s) [upload=463344]: Upload finished. err: <nil> Duration: 50.655401963s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)