Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X.............X......] duration: 5.835352394s trace: https://jaeger.dev.storj.io/trace/512bcb9bf08d2da9 FAIL (2026-01-12 18:57:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 12 18:52:59.379576882 (915h23m59.250659917s) [upload=801199]: Starting upload Jan 12 18:52:59.379630744 (915h23m59.250713779s) [upload=801199]: Done starting upload Jan 12 18:52:59.379719844 (915h23m59.250802879s) [upload=801199]: Uploading segments... Jan 12 18:52:59.379879347 (915h23m59.250962382s) [upload=801199]: Got next segment. Inline: false Jan 12 18:52:59.379917681 (915h23m59.251000726s) [upload=801199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 18:52:59.379922670 (915h23m59.251005716s) [upload=801199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 18:52:59.945729741 (915h23m59.816812776s) [seg_pos={0 0}, upload=801199]: Begin upload segment... Jan 12 18:52:59.945777812 (915h23m59.816860847s) [seg_pos={0 0}, upload=801199]: Done begin upload segment. Jan 12 18:52:59.945872483 (915h23m59.816955518s) [noise=true , address=145.79.227.20:4439 , piece_id=R76OUH3NPZFJMHP7C5YLH44RLX2J7VIC4R3EHUZFTMCK7BAKPNJQ, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.945889696 (915h23m59.816972731s) [noise=true , address=77.90.31.21:4880 , piece_id=IIHNAKYPKV3DIUNZOCGK3ZVOYQXFYWEKMCWTULKYM76J2ZXRC53A, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.945974337 (915h23m59.817057372s) [noise=true , address=45.91.116.9:4262 , piece_id=LNLGQ3ZKC4FSXMLLUGKM7RMDVR5BIDLUALFH25QQV2TCFHK6VFQA, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.946516238 (915h23m59.817599273s) [noise=true , address=77.90.31.12:4590 , piece_id=3EW5KXCEXD37TTKUAMO3OJEQEZGYGCVDRWCKUC7P7KPGMJUVCOVA, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.946888608 (915h23m59.817971643s) [noise=true , address=77.90.31.11:4555 , piece_id=DFQXR5X3OCZIVCZSESGUL3GK2F6E7EKVHMOOI5VDOM452ERKIN6Q, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.946895791 (915h23m59.817978826s) [noise=true , address=77.90.31.18:4808 , piece_id=R2VL26Y4B2C2IVLT4OZ5F3U2HGIIO75IIEO24QUAGSNQRCA7VC3A, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.947217193 (915h23m59.818300228s) [noise=true , address=45.91.116.6:4973 , piece_id=O7HE3VVAJRQAO4B52YR3YQ5OBGMZAKDRFHYLVKTYI46QAT5NVRYQ, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.947275544 (915h23m59.818358579s) [noise=true , address=145.79.227.7:4295 , piece_id=NK4344JKJ6PFU6T2NY62MQBYILLI52ITNI2FDO36MRY64CCZEQTA, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.947584833 (915h23m59.818667868s) [noise=true , address=145.79.227.45:4231 , piece_id=O36XJLX6OBD4YFYLNNWSXCJI3K5WZMLFNDPYASYSGSGWY6DJSWRQ, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.947599621 (915h23m59.818682656s) [noise=true , address=77.90.31.10:4519 , piece_id=2PNRYI7FPX7UMRI4FAQWA3TMITJ5SESOPKJF5U4Q6WEMPACR6NRQ, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.947961120 (915h23m59.819044155s) [noise=true , address=45.91.116.12:4227 , piece_id=OFOCOYEFOAZGKJOFWVNLYM6FDKRWMNNSWR7UCGFNFCX7OIG3XH4Q, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.948035271 (915h23m59.819118316s) [noise=true , address=145.79.227.11:4731 , piece_id=LYMJBPG2ZPATR3UNXLLDC5SY22EOVC6YBCEQIEE3XGZ4RJRSJXZQ, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.948293923 (915h23m59.819376958s) [noise=true , address=45.91.116.8:4047 , piece_id=TYFKNAOCLKSQBEUOX7JCUAVECLUSOJMZL3BQNWOX7QK7WRRIEI6Q, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.948360670 (915h23m59.819443705s) [noise=true , address=145.79.227.52:4446 , piece_id=2JRS3WPJ2PAFE3B3EUZO27DQS4HDKPHXZYIGIWFJTDWRGSV3IRIA, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.948598854 (915h23m59.819681889s) [noise=true , address=145.79.227.16:4910 , piece_id=I2O3LBJLA3NUICDDB52PRRQWXZWE7VE2F73C5TZ6G5CPQ55EI47A, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.948693183 (915h23m59.819776218s) [noise=true , address=45.91.116.5:4300 , piece_id=ABVDLNQ7ZD7M3RVJLCZNKAHWU5MVJTKVLHJLERRXCB4ZVZHELSCA, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:53:01.738921723 (915h24m1.610004768s) [upload=801199]: Next returned nil segment Jan 12 18:53:01.738939648 (915h24m1.610022693s) [upload=801199]: Waiting for error group managing segments... Jan 12 18:53:02.073911963 (915h24m1.944995008s) [noise=true , address=77.90.31.10:4519 , piece_id=2PNRYI7FPX7UMRI4FAQWA3TMITJ5SESOPKJF5U4Q6WEMPACR6NRQ, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.081381215 (915h24m1.95246426s) [noise=true , address=145.79.227.16:4910 , piece_id=I2O3LBJLA3NUICDDB52PRRQWXZWE7VE2F73C5TZ6G5CPQ55EI47A, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.089752244 (915h24m1.960835289s) [noise=true , address=77.90.31.11:4555 , piece_id=DFQXR5X3OCZIVCZSESGUL3GK2F6E7EKVHMOOI5VDOM452ERKIN6Q, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.099125060 (915h24m1.970208105s) [noise=true , address=77.90.31.12:4590 , piece_id=3EW5KXCEXD37TTKUAMO3OJEQEZGYGCVDRWCKUC7P7KPGMJUVCOVA, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.119372481 (915h24m1.990455526s) [noise=true , address=77.90.31.21:4880 , piece_id=IIHNAKYPKV3DIUNZOCGK3ZVOYQXFYWEKMCWTULKYM76J2ZXRC53A, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.160786966 (915h24m2.031870011s) [noise=true , address=145.79.227.7:4295 , piece_id=NK4344JKJ6PFU6T2NY62MQBYILLI52ITNI2FDO36MRY64CCZEQTA, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.173398348 (915h24m2.044481383s) [noise=true , address=77.90.31.18:4808 , piece_id=R2VL26Y4B2C2IVLT4OZ5F3U2HGIIO75IIEO24QUAGSNQRCA7VC3A, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.191338316 (915h24m2.062421361s) [noise=true , address=145.79.227.52:4446 , piece_id=2JRS3WPJ2PAFE3B3EUZO27DQS4HDKPHXZYIGIWFJTDWRGSV3IRIA, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.193169993 (915h24m2.064253038s) [noise=true , address=45.91.116.12:4227 , piece_id=OFOCOYEFOAZGKJOFWVNLYM6FDKRWMNNSWR7UCGFNFCX7OIG3XH4Q, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.207190219 (915h24m2.078273254s) [noise=true , address=45.91.116.9:4262 , piece_id=LNLGQ3ZKC4FSXMLLUGKM7RMDVR5BIDLUALFH25QQV2TCFHK6VFQA, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.235451699 (915h24m2.106534744s) [noise=true , address=45.91.116.6:4973 , piece_id=O7HE3VVAJRQAO4B52YR3YQ5OBGMZAKDRFHYLVKTYI46QAT5NVRYQ, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.264976344 (915h24m2.136059389s) [noise=true , address=145.79.227.11:4731 , piece_id=LYMJBPG2ZPATR3UNXLLDC5SY22EOVC6YBCEQIEE3XGZ4RJRSJXZQ, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.278399723 (915h24m2.149482768s) [noise=true , address=145.79.227.45:4231 , piece_id=O36XJLX6OBD4YFYLNNWSXCJI3K5WZMLFNDPYASYSGSGWY6DJSWRQ, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.278424661 (915h24m2.149507696s) [seg_pos={0 0}, upload=801199]: Segment reached optimal threshold of 13 pieces. Jan 12 18:53:02.278758596 (915h24m2.149841671s) [noise=true , address=45.91.116.5:4300 , piece_id=ABVDLNQ7ZD7M3RVJLCZNKAHWU5MVJTKVLHJLERRXCB4ZVZHELSCA, seg_pos={0 0}, upload=801199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 12 18:53:02.278800566 (915h24m2.149883621s) [noise=true , address=45.91.116.5:4300 , piece_id=ABVDLNQ7ZD7M3RVJLCZNKAHWU5MVJTKVLHJLERRXCB4ZVZHELSCA, seg_pos={0 0}, upload=801199]: Long tail cancellation due to optimal threshold Jan 12 18:53:02.278925184 (915h24m2.150008229s) [noise=true , address=145.79.227.20:4439 , piece_id=R76OUH3NPZFJMHP7C5YLH44RLX2J7VIC4R3EHUZFTMCK7BAKPNJQ, seg_pos={0 0}, upload=801199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn): protocol: expected piece hash; context canceled; EOF Jan 12 18:53:02.278936645 (915h24m2.1500197s) [noise=true , address=145.79.227.20:4439 , piece_id=R76OUH3NPZFJMHP7C5YLH44RLX2J7VIC4R3EHUZFTMCK7BAKPNJQ, seg_pos={0 0}, upload=801199]: Long tail cancellation due to optimal threshold Jan 12 18:53:02.279000747 (915h24m2.150083792s) [noise=true , address=45.91.116.8:4047 , piece_id=TYFKNAOCLKSQBEUOX7JCUAVECLUSOJMZL3BQNWOX7QK7WRRIEI6Q, seg_pos={0 0}, upload=801199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:56336->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:56336->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56336->45.91.116.8:4047: use of closed network connection Jan 12 18:53:02.279008672 (915h24m2.150091717s) [noise=true , address=45.91.116.8:4047 , piece_id=TYFKNAOCLKSQBEUOX7JCUAVECLUSOJMZL3BQNWOX7QK7WRRIEI6Q, seg_pos={0 0}, upload=801199]: Long tail cancellation due to optimal threshold Jan 12 18:53:02.279037688 (915h24m2.150120723s) [seg_pos={0 0}, upload=801199]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 18:53:02.279149039 (915h24m2.150232115s) [upload=801199]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 18:53:02.279156413 (915h24m2.150239468s) [upload=801199]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 18:53:02.601181307 (915h24m2.472264352s) [upload=801199]: Done uploading segments... Jan 12 18:53:02.601202908 (915h24m2.472285953s) [upload=801199]: Upload finished. err: <nil> Duration: 5.835377011s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [X.X.XXX.X.XXXXX.XX....X.XXX.X..X.X.X....XX...........XX....X] duration: 5.001533754s trace: https://jaeger.dev.storj.io/trace/3dd5d516d7d8c6dc FAIL (2026-01-12 18:56:45 UTC) orbiter warning: context deadline exceeded Duration: 5.001555164s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X.............X.....] duration: 5.849210671s trace: https://jaeger.dev.storj.io/trace/512bcb9bf08d2da9 FAIL (2026-01-12 18:56:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 12 18:52:59.379576882 (915h23m59.250659917s) [upload=801199]: Starting upload Jan 12 18:52:59.379630744 (915h23m59.250713779s) [upload=801199]: Done starting upload Jan 12 18:52:59.379719844 (915h23m59.250802879s) [upload=801199]: Uploading segments... Jan 12 18:52:59.379879347 (915h23m59.250962382s) [upload=801199]: Got next segment. Inline: false Jan 12 18:52:59.379917681 (915h23m59.251000726s) [upload=801199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 18:52:59.379922670 (915h23m59.251005716s) [upload=801199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 18:52:59.945729741 (915h23m59.816812776s) [seg_pos={0 0}, upload=801199]: Begin upload segment... Jan 12 18:52:59.945777812 (915h23m59.816860847s) [seg_pos={0 0}, upload=801199]: Done begin upload segment. Jan 12 18:52:59.945872483 (915h23m59.816955518s) [noise=true , address=145.79.227.20:4439 , piece_id=R76OUH3NPZFJMHP7C5YLH44RLX2J7VIC4R3EHUZFTMCK7BAKPNJQ, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.945889696 (915h23m59.816972731s) [noise=true , address=77.90.31.21:4880 , piece_id=IIHNAKYPKV3DIUNZOCGK3ZVOYQXFYWEKMCWTULKYM76J2ZXRC53A, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.945974337 (915h23m59.817057372s) [noise=true , address=45.91.116.9:4262 , piece_id=LNLGQ3ZKC4FSXMLLUGKM7RMDVR5BIDLUALFH25QQV2TCFHK6VFQA, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.946516238 (915h23m59.817599273s) [noise=true , address=77.90.31.12:4590 , piece_id=3EW5KXCEXD37TTKUAMO3OJEQEZGYGCVDRWCKUC7P7KPGMJUVCOVA, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.946888608 (915h23m59.817971643s) [noise=true , address=77.90.31.11:4555 , piece_id=DFQXR5X3OCZIVCZSESGUL3GK2F6E7EKVHMOOI5VDOM452ERKIN6Q, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.946895791 (915h23m59.817978826s) [noise=true , address=77.90.31.18:4808 , piece_id=R2VL26Y4B2C2IVLT4OZ5F3U2HGIIO75IIEO24QUAGSNQRCA7VC3A, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.947217193 (915h23m59.818300228s) [noise=true , address=45.91.116.6:4973 , piece_id=O7HE3VVAJRQAO4B52YR3YQ5OBGMZAKDRFHYLVKTYI46QAT5NVRYQ, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.947275544 (915h23m59.818358579s) [noise=true , address=145.79.227.7:4295 , piece_id=NK4344JKJ6PFU6T2NY62MQBYILLI52ITNI2FDO36MRY64CCZEQTA, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.947584833 (915h23m59.818667868s) [noise=true , address=145.79.227.45:4231 , piece_id=O36XJLX6OBD4YFYLNNWSXCJI3K5WZMLFNDPYASYSGSGWY6DJSWRQ, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.947599621 (915h23m59.818682656s) [noise=true , address=77.90.31.10:4519 , piece_id=2PNRYI7FPX7UMRI4FAQWA3TMITJ5SESOPKJF5U4Q6WEMPACR6NRQ, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.947961120 (915h23m59.819044155s) [noise=true , address=45.91.116.12:4227 , piece_id=OFOCOYEFOAZGKJOFWVNLYM6FDKRWMNNSWR7UCGFNFCX7OIG3XH4Q, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.948035271 (915h23m59.819118316s) [noise=true , address=145.79.227.11:4731 , piece_id=LYMJBPG2ZPATR3UNXLLDC5SY22EOVC6YBCEQIEE3XGZ4RJRSJXZQ, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.948293923 (915h23m59.819376958s) [noise=true , address=45.91.116.8:4047 , piece_id=TYFKNAOCLKSQBEUOX7JCUAVECLUSOJMZL3BQNWOX7QK7WRRIEI6Q, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.948360670 (915h23m59.819443705s) [noise=true , address=145.79.227.52:4446 , piece_id=2JRS3WPJ2PAFE3B3EUZO27DQS4HDKPHXZYIGIWFJTDWRGSV3IRIA, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.948598854 (915h23m59.819681889s) [noise=true , address=145.79.227.16:4910 , piece_id=I2O3LBJLA3NUICDDB52PRRQWXZWE7VE2F73C5TZ6G5CPQ55EI47A, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.948693183 (915h23m59.819776218s) [noise=true , address=45.91.116.5:4300 , piece_id=ABVDLNQ7ZD7M3RVJLCZNKAHWU5MVJTKVLHJLERRXCB4ZVZHELSCA, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:53:01.738921723 (915h24m1.610004768s) [upload=801199]: Next returned nil segment Jan 12 18:53:01.738939648 (915h24m1.610022693s) [upload=801199]: Waiting for error group managing segments... Jan 12 18:53:02.073911963 (915h24m1.944995008s) [noise=true , address=77.90.31.10:4519 , piece_id=2PNRYI7FPX7UMRI4FAQWA3TMITJ5SESOPKJF5U4Q6WEMPACR6NRQ, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.081381215 (915h24m1.95246426s) [noise=true , address=145.79.227.16:4910 , piece_id=I2O3LBJLA3NUICDDB52PRRQWXZWE7VE2F73C5TZ6G5CPQ55EI47A, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.089752244 (915h24m1.960835289s) [noise=true , address=77.90.31.11:4555 , piece_id=DFQXR5X3OCZIVCZSESGUL3GK2F6E7EKVHMOOI5VDOM452ERKIN6Q, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.099125060 (915h24m1.970208105s) [noise=true , address=77.90.31.12:4590 , piece_id=3EW5KXCEXD37TTKUAMO3OJEQEZGYGCVDRWCKUC7P7KPGMJUVCOVA, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.119372481 (915h24m1.990455526s) [noise=true , address=77.90.31.21:4880 , piece_id=IIHNAKYPKV3DIUNZOCGK3ZVOYQXFYWEKMCWTULKYM76J2ZXRC53A, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.160786966 (915h24m2.031870011s) [noise=true , address=145.79.227.7:4295 , piece_id=NK4344JKJ6PFU6T2NY62MQBYILLI52ITNI2FDO36MRY64CCZEQTA, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.173398348 (915h24m2.044481383s) [noise=true , address=77.90.31.18:4808 , piece_id=R2VL26Y4B2C2IVLT4OZ5F3U2HGIIO75IIEO24QUAGSNQRCA7VC3A, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.191338316 (915h24m2.062421361s) [noise=true , address=145.79.227.52:4446 , piece_id=2JRS3WPJ2PAFE3B3EUZO27DQS4HDKPHXZYIGIWFJTDWRGSV3IRIA, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.193169993 (915h24m2.064253038s) [noise=true , address=45.91.116.12:4227 , piece_id=OFOCOYEFOAZGKJOFWVNLYM6FDKRWMNNSWR7UCGFNFCX7OIG3XH4Q, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.207190219 (915h24m2.078273254s) [noise=true , address=45.91.116.9:4262 , piece_id=LNLGQ3ZKC4FSXMLLUGKM7RMDVR5BIDLUALFH25QQV2TCFHK6VFQA, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.235451699 (915h24m2.106534744s) [noise=true , address=45.91.116.6:4973 , piece_id=O7HE3VVAJRQAO4B52YR3YQ5OBGMZAKDRFHYLVKTYI46QAT5NVRYQ, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.264976344 (915h24m2.136059389s) [noise=true , address=145.79.227.11:4731 , piece_id=LYMJBPG2ZPATR3UNXLLDC5SY22EOVC6YBCEQIEE3XGZ4RJRSJXZQ, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.278399723 (915h24m2.149482768s) [noise=true , address=145.79.227.45:4231 , piece_id=O36XJLX6OBD4YFYLNNWSXCJI3K5WZMLFNDPYASYSGSGWY6DJSWRQ, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.278424661 (915h24m2.149507696s) [seg_pos={0 0}, upload=801199]: Segment reached optimal threshold of 13 pieces. Jan 12 18:53:02.278758596 (915h24m2.149841671s) [noise=true , address=45.91.116.5:4300 , piece_id=ABVDLNQ7ZD7M3RVJLCZNKAHWU5MVJTKVLHJLERRXCB4ZVZHELSCA, seg_pos={0 0}, upload=801199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 12 18:53:02.278800566 (915h24m2.149883621s) [noise=true , address=45.91.116.5:4300 , piece_id=ABVDLNQ7ZD7M3RVJLCZNKAHWU5MVJTKVLHJLERRXCB4ZVZHELSCA, seg_pos={0 0}, upload=801199]: Long tail cancellation due to optimal threshold Jan 12 18:53:02.278925184 (915h24m2.150008229s) [noise=true , address=145.79.227.20:4439 , piece_id=R76OUH3NPZFJMHP7C5YLH44RLX2J7VIC4R3EHUZFTMCK7BAKPNJQ, seg_pos={0 0}, upload=801199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn): protocol: expected piece hash; context canceled; EOF Jan 12 18:53:02.278936645 (915h24m2.1500197s) [noise=true , address=145.79.227.20:4439 , piece_id=R76OUH3NPZFJMHP7C5YLH44RLX2J7VIC4R3EHUZFTMCK7BAKPNJQ, seg_pos={0 0}, upload=801199]: Long tail cancellation due to optimal threshold Jan 12 18:53:02.279000747 (915h24m2.150083792s) [noise=true , address=45.91.116.8:4047 , piece_id=TYFKNAOCLKSQBEUOX7JCUAVECLUSOJMZL3BQNWOX7QK7WRRIEI6Q, seg_pos={0 0}, upload=801199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:56336->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:56336->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56336->45.91.116.8:4047: use of closed network connection Jan 12 18:53:02.279008672 (915h24m2.150091717s) [noise=true , address=45.91.116.8:4047 , piece_id=TYFKNAOCLKSQBEUOX7JCUAVECLUSOJMZL3BQNWOX7QK7WRRIEI6Q, seg_pos={0 0}, upload=801199]: Long tail cancellation due to optimal threshold Jan 12 18:53:02.279037688 (915h24m2.150120723s) [seg_pos={0 0}, upload=801199]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 18:53:02.279149039 (915h24m2.150232115s) [upload=801199]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 18:53:02.279156413 (915h24m2.150239468s) [upload=801199]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 18:53:02.601181307 (915h24m2.472264352s) [upload=801199]: Done uploading segments... Jan 12 18:53:02.601202908 (915h24m2.472285953s) [upload=801199]: Upload finished. err: <nil> Duration: 5.849239627s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 26 out of 60 is failed (threshold=6) [XX.X.XXX.X.XXXXX.XX....X.XXX.X..X.X.X....XX...........XX....] duration: 3.581960881s trace: https://jaeger.dev.storj.io/trace/70bcd4df5d87a249 FAIL (2026-01-12 18:56:10 UTC) orbiter warning: context deadline exceeded Duration: 3.58198724s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X.............X....] duration: 5.807542898s trace: https://jaeger.dev.storj.io/trace/512bcb9bf08d2da9 FAIL (2026-01-12 18:55:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 12 18:52:59.379576882 (915h23m59.250659917s) [upload=801199]: Starting upload Jan 12 18:52:59.379630744 (915h23m59.250713779s) [upload=801199]: Done starting upload Jan 12 18:52:59.379719844 (915h23m59.250802879s) [upload=801199]: Uploading segments... Jan 12 18:52:59.379879347 (915h23m59.250962382s) [upload=801199]: Got next segment. Inline: false Jan 12 18:52:59.379917681 (915h23m59.251000726s) [upload=801199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 18:52:59.379922670 (915h23m59.251005716s) [upload=801199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 18:52:59.945729741 (915h23m59.816812776s) [seg_pos={0 0}, upload=801199]: Begin upload segment... Jan 12 18:52:59.945777812 (915h23m59.816860847s) [seg_pos={0 0}, upload=801199]: Done begin upload segment. Jan 12 18:52:59.945872483 (915h23m59.816955518s) [noise=true , address=145.79.227.20:4439 , piece_id=R76OUH3NPZFJMHP7C5YLH44RLX2J7VIC4R3EHUZFTMCK7BAKPNJQ, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.945889696 (915h23m59.816972731s) [noise=true , address=77.90.31.21:4880 , piece_id=IIHNAKYPKV3DIUNZOCGK3ZVOYQXFYWEKMCWTULKYM76J2ZXRC53A, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.945974337 (915h23m59.817057372s) [noise=true , address=45.91.116.9:4262 , piece_id=LNLGQ3ZKC4FSXMLLUGKM7RMDVR5BIDLUALFH25QQV2TCFHK6VFQA, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.946516238 (915h23m59.817599273s) [noise=true , address=77.90.31.12:4590 , piece_id=3EW5KXCEXD37TTKUAMO3OJEQEZGYGCVDRWCKUC7P7KPGMJUVCOVA, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.946888608 (915h23m59.817971643s) [noise=true , address=77.90.31.11:4555 , piece_id=DFQXR5X3OCZIVCZSESGUL3GK2F6E7EKVHMOOI5VDOM452ERKIN6Q, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.946895791 (915h23m59.817978826s) [noise=true , address=77.90.31.18:4808 , piece_id=R2VL26Y4B2C2IVLT4OZ5F3U2HGIIO75IIEO24QUAGSNQRCA7VC3A, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.947217193 (915h23m59.818300228s) [noise=true , address=45.91.116.6:4973 , piece_id=O7HE3VVAJRQAO4B52YR3YQ5OBGMZAKDRFHYLVKTYI46QAT5NVRYQ, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.947275544 (915h23m59.818358579s) [noise=true , address=145.79.227.7:4295 , piece_id=NK4344JKJ6PFU6T2NY62MQBYILLI52ITNI2FDO36MRY64CCZEQTA, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.947584833 (915h23m59.818667868s) [noise=true , address=145.79.227.45:4231 , piece_id=O36XJLX6OBD4YFYLNNWSXCJI3K5WZMLFNDPYASYSGSGWY6DJSWRQ, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.947599621 (915h23m59.818682656s) [noise=true , address=77.90.31.10:4519 , piece_id=2PNRYI7FPX7UMRI4FAQWA3TMITJ5SESOPKJF5U4Q6WEMPACR6NRQ, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.947961120 (915h23m59.819044155s) [noise=true , address=45.91.116.12:4227 , piece_id=OFOCOYEFOAZGKJOFWVNLYM6FDKRWMNNSWR7UCGFNFCX7OIG3XH4Q, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.948035271 (915h23m59.819118316s) [noise=true , address=145.79.227.11:4731 , piece_id=LYMJBPG2ZPATR3UNXLLDC5SY22EOVC6YBCEQIEE3XGZ4RJRSJXZQ, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.948293923 (915h23m59.819376958s) [noise=true , address=45.91.116.8:4047 , piece_id=TYFKNAOCLKSQBEUOX7JCUAVECLUSOJMZL3BQNWOX7QK7WRRIEI6Q, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.948360670 (915h23m59.819443705s) [noise=true , address=145.79.227.52:4446 , piece_id=2JRS3WPJ2PAFE3B3EUZO27DQS4HDKPHXZYIGIWFJTDWRGSV3IRIA, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.948598854 (915h23m59.819681889s) [noise=true , address=145.79.227.16:4910 , piece_id=I2O3LBJLA3NUICDDB52PRRQWXZWE7VE2F73C5TZ6G5CPQ55EI47A, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.948693183 (915h23m59.819776218s) [noise=true , address=45.91.116.5:4300 , piece_id=ABVDLNQ7ZD7M3RVJLCZNKAHWU5MVJTKVLHJLERRXCB4ZVZHELSCA, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:53:01.738921723 (915h24m1.610004768s) [upload=801199]: Next returned nil segment Jan 12 18:53:01.738939648 (915h24m1.610022693s) [upload=801199]: Waiting for error group managing segments... Jan 12 18:53:02.073911963 (915h24m1.944995008s) [noise=true , address=77.90.31.10:4519 , piece_id=2PNRYI7FPX7UMRI4FAQWA3TMITJ5SESOPKJF5U4Q6WEMPACR6NRQ, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.081381215 (915h24m1.95246426s) [noise=true , address=145.79.227.16:4910 , piece_id=I2O3LBJLA3NUICDDB52PRRQWXZWE7VE2F73C5TZ6G5CPQ55EI47A, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.089752244 (915h24m1.960835289s) [noise=true , address=77.90.31.11:4555 , piece_id=DFQXR5X3OCZIVCZSESGUL3GK2F6E7EKVHMOOI5VDOM452ERKIN6Q, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.099125060 (915h24m1.970208105s) [noise=true , address=77.90.31.12:4590 , piece_id=3EW5KXCEXD37TTKUAMO3OJEQEZGYGCVDRWCKUC7P7KPGMJUVCOVA, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.119372481 (915h24m1.990455526s) [noise=true , address=77.90.31.21:4880 , piece_id=IIHNAKYPKV3DIUNZOCGK3ZVOYQXFYWEKMCWTULKYM76J2ZXRC53A, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.160786966 (915h24m2.031870011s) [noise=true , address=145.79.227.7:4295 , piece_id=NK4344JKJ6PFU6T2NY62MQBYILLI52ITNI2FDO36MRY64CCZEQTA, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.173398348 (915h24m2.044481383s) [noise=true , address=77.90.31.18:4808 , piece_id=R2VL26Y4B2C2IVLT4OZ5F3U2HGIIO75IIEO24QUAGSNQRCA7VC3A, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.191338316 (915h24m2.062421361s) [noise=true , address=145.79.227.52:4446 , piece_id=2JRS3WPJ2PAFE3B3EUZO27DQS4HDKPHXZYIGIWFJTDWRGSV3IRIA, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.193169993 (915h24m2.064253038s) [noise=true , address=45.91.116.12:4227 , piece_id=OFOCOYEFOAZGKJOFWVNLYM6FDKRWMNNSWR7UCGFNFCX7OIG3XH4Q, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.207190219 (915h24m2.078273254s) [noise=true , address=45.91.116.9:4262 , piece_id=LNLGQ3ZKC4FSXMLLUGKM7RMDVR5BIDLUALFH25QQV2TCFHK6VFQA, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.235451699 (915h24m2.106534744s) [noise=true , address=45.91.116.6:4973 , piece_id=O7HE3VVAJRQAO4B52YR3YQ5OBGMZAKDRFHYLVKTYI46QAT5NVRYQ, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.264976344 (915h24m2.136059389s) [noise=true , address=145.79.227.11:4731 , piece_id=LYMJBPG2ZPATR3UNXLLDC5SY22EOVC6YBCEQIEE3XGZ4RJRSJXZQ, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.278399723 (915h24m2.149482768s) [noise=true , address=145.79.227.45:4231 , piece_id=O36XJLX6OBD4YFYLNNWSXCJI3K5WZMLFNDPYASYSGSGWY6DJSWRQ, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.278424661 (915h24m2.149507696s) [seg_pos={0 0}, upload=801199]: Segment reached optimal threshold of 13 pieces. Jan 12 18:53:02.278758596 (915h24m2.149841671s) [noise=true , address=45.91.116.5:4300 , piece_id=ABVDLNQ7ZD7M3RVJLCZNKAHWU5MVJTKVLHJLERRXCB4ZVZHELSCA, seg_pos={0 0}, upload=801199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 12 18:53:02.278800566 (915h24m2.149883621s) [noise=true , address=45.91.116.5:4300 , piece_id=ABVDLNQ7ZD7M3RVJLCZNKAHWU5MVJTKVLHJLERRXCB4ZVZHELSCA, seg_pos={0 0}, upload=801199]: Long tail cancellation due to optimal threshold Jan 12 18:53:02.278925184 (915h24m2.150008229s) [noise=true , address=145.79.227.20:4439 , piece_id=R76OUH3NPZFJMHP7C5YLH44RLX2J7VIC4R3EHUZFTMCK7BAKPNJQ, seg_pos={0 0}, upload=801199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn): protocol: expected piece hash; context canceled; EOF Jan 12 18:53:02.278936645 (915h24m2.1500197s) [noise=true , address=145.79.227.20:4439 , piece_id=R76OUH3NPZFJMHP7C5YLH44RLX2J7VIC4R3EHUZFTMCK7BAKPNJQ, seg_pos={0 0}, upload=801199]: Long tail cancellation due to optimal threshold Jan 12 18:53:02.279000747 (915h24m2.150083792s) [noise=true , address=45.91.116.8:4047 , piece_id=TYFKNAOCLKSQBEUOX7JCUAVECLUSOJMZL3BQNWOX7QK7WRRIEI6Q, seg_pos={0 0}, upload=801199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:56336->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:56336->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56336->45.91.116.8:4047: use of closed network connection Jan 12 18:53:02.279008672 (915h24m2.150091717s) [noise=true , address=45.91.116.8:4047 , piece_id=TYFKNAOCLKSQBEUOX7JCUAVECLUSOJMZL3BQNWOX7QK7WRRIEI6Q, seg_pos={0 0}, upload=801199]: Long tail cancellation due to optimal threshold Jan 12 18:53:02.279037688 (915h24m2.150120723s) [seg_pos={0 0}, upload=801199]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 18:53:02.279149039 (915h24m2.150232115s) [upload=801199]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 18:53:02.279156413 (915h24m2.150239468s) [upload=801199]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 18:53:02.601181307 (915h24m2.472264352s) [upload=801199]: Done uploading segments... Jan 12 18:53:02.601202908 (915h24m2.472285953s) [upload=801199]: Upload finished. err: <nil> Duration: 5.807568185s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 27 out of 60 is failed (threshold=6) [XXX.X.XXX.X.XXXXX.XX....X.XXX.X..X.X.X....XX...........XX...] duration: 3.703931377s trace: https://jaeger.dev.storj.io/trace/70bcd4df5d87a249 FAIL (2026-01-12 18:55:16 UTC) orbiter warning: context deadline exceeded Duration: 3.703956334s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X.............X...] duration: 5.81166842s trace: https://jaeger.dev.storj.io/trace/512bcb9bf08d2da9 FAIL (2026-01-12 18:55:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 12 18:52:59.379576882 (915h23m59.250659917s) [upload=801199]: Starting upload Jan 12 18:52:59.379630744 (915h23m59.250713779s) [upload=801199]: Done starting upload Jan 12 18:52:59.379719844 (915h23m59.250802879s) [upload=801199]: Uploading segments... Jan 12 18:52:59.379879347 (915h23m59.250962382s) [upload=801199]: Got next segment. Inline: false Jan 12 18:52:59.379917681 (915h23m59.251000726s) [upload=801199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 18:52:59.379922670 (915h23m59.251005716s) [upload=801199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 18:52:59.945729741 (915h23m59.816812776s) [seg_pos={0 0}, upload=801199]: Begin upload segment... Jan 12 18:52:59.945777812 (915h23m59.816860847s) [seg_pos={0 0}, upload=801199]: Done begin upload segment. Jan 12 18:52:59.945872483 (915h23m59.816955518s) [noise=true , address=145.79.227.20:4439 , piece_id=R76OUH3NPZFJMHP7C5YLH44RLX2J7VIC4R3EHUZFTMCK7BAKPNJQ, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.945889696 (915h23m59.816972731s) [noise=true , address=77.90.31.21:4880 , piece_id=IIHNAKYPKV3DIUNZOCGK3ZVOYQXFYWEKMCWTULKYM76J2ZXRC53A, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.945974337 (915h23m59.817057372s) [noise=true , address=45.91.116.9:4262 , piece_id=LNLGQ3ZKC4FSXMLLUGKM7RMDVR5BIDLUALFH25QQV2TCFHK6VFQA, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.946516238 (915h23m59.817599273s) [noise=true , address=77.90.31.12:4590 , piece_id=3EW5KXCEXD37TTKUAMO3OJEQEZGYGCVDRWCKUC7P7KPGMJUVCOVA, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.946888608 (915h23m59.817971643s) [noise=true , address=77.90.31.11:4555 , piece_id=DFQXR5X3OCZIVCZSESGUL3GK2F6E7EKVHMOOI5VDOM452ERKIN6Q, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.946895791 (915h23m59.817978826s) [noise=true , address=77.90.31.18:4808 , piece_id=R2VL26Y4B2C2IVLT4OZ5F3U2HGIIO75IIEO24QUAGSNQRCA7VC3A, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.947217193 (915h23m59.818300228s) [noise=true , address=45.91.116.6:4973 , piece_id=O7HE3VVAJRQAO4B52YR3YQ5OBGMZAKDRFHYLVKTYI46QAT5NVRYQ, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.947275544 (915h23m59.818358579s) [noise=true , address=145.79.227.7:4295 , piece_id=NK4344JKJ6PFU6T2NY62MQBYILLI52ITNI2FDO36MRY64CCZEQTA, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.947584833 (915h23m59.818667868s) [noise=true , address=145.79.227.45:4231 , piece_id=O36XJLX6OBD4YFYLNNWSXCJI3K5WZMLFNDPYASYSGSGWY6DJSWRQ, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.947599621 (915h23m59.818682656s) [noise=true , address=77.90.31.10:4519 , piece_id=2PNRYI7FPX7UMRI4FAQWA3TMITJ5SESOPKJF5U4Q6WEMPACR6NRQ, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.947961120 (915h23m59.819044155s) [noise=true , address=45.91.116.12:4227 , piece_id=OFOCOYEFOAZGKJOFWVNLYM6FDKRWMNNSWR7UCGFNFCX7OIG3XH4Q, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.948035271 (915h23m59.819118316s) [noise=true , address=145.79.227.11:4731 , piece_id=LYMJBPG2ZPATR3UNXLLDC5SY22EOVC6YBCEQIEE3XGZ4RJRSJXZQ, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.948293923 (915h23m59.819376958s) [noise=true , address=45.91.116.8:4047 , piece_id=TYFKNAOCLKSQBEUOX7JCUAVECLUSOJMZL3BQNWOX7QK7WRRIEI6Q, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.948360670 (915h23m59.819443705s) [noise=true , address=145.79.227.52:4446 , piece_id=2JRS3WPJ2PAFE3B3EUZO27DQS4HDKPHXZYIGIWFJTDWRGSV3IRIA, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.948598854 (915h23m59.819681889s) [noise=true , address=145.79.227.16:4910 , piece_id=I2O3LBJLA3NUICDDB52PRRQWXZWE7VE2F73C5TZ6G5CPQ55EI47A, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.948693183 (915h23m59.819776218s) [noise=true , address=45.91.116.5:4300 , piece_id=ABVDLNQ7ZD7M3RVJLCZNKAHWU5MVJTKVLHJLERRXCB4ZVZHELSCA, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:53:01.738921723 (915h24m1.610004768s) [upload=801199]: Next returned nil segment Jan 12 18:53:01.738939648 (915h24m1.610022693s) [upload=801199]: Waiting for error group managing segments... Jan 12 18:53:02.073911963 (915h24m1.944995008s) [noise=true , address=77.90.31.10:4519 , piece_id=2PNRYI7FPX7UMRI4FAQWA3TMITJ5SESOPKJF5U4Q6WEMPACR6NRQ, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.081381215 (915h24m1.95246426s) [noise=true , address=145.79.227.16:4910 , piece_id=I2O3LBJLA3NUICDDB52PRRQWXZWE7VE2F73C5TZ6G5CPQ55EI47A, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.089752244 (915h24m1.960835289s) [noise=true , address=77.90.31.11:4555 , piece_id=DFQXR5X3OCZIVCZSESGUL3GK2F6E7EKVHMOOI5VDOM452ERKIN6Q, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.099125060 (915h24m1.970208105s) [noise=true , address=77.90.31.12:4590 , piece_id=3EW5KXCEXD37TTKUAMO3OJEQEZGYGCVDRWCKUC7P7KPGMJUVCOVA, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.119372481 (915h24m1.990455526s) [noise=true , address=77.90.31.21:4880 , piece_id=IIHNAKYPKV3DIUNZOCGK3ZVOYQXFYWEKMCWTULKYM76J2ZXRC53A, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.160786966 (915h24m2.031870011s) [noise=true , address=145.79.227.7:4295 , piece_id=NK4344JKJ6PFU6T2NY62MQBYILLI52ITNI2FDO36MRY64CCZEQTA, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.173398348 (915h24m2.044481383s) [noise=true , address=77.90.31.18:4808 , piece_id=R2VL26Y4B2C2IVLT4OZ5F3U2HGIIO75IIEO24QUAGSNQRCA7VC3A, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.191338316 (915h24m2.062421361s) [noise=true , address=145.79.227.52:4446 , piece_id=2JRS3WPJ2PAFE3B3EUZO27DQS4HDKPHXZYIGIWFJTDWRGSV3IRIA, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.193169993 (915h24m2.064253038s) [noise=true , address=45.91.116.12:4227 , piece_id=OFOCOYEFOAZGKJOFWVNLYM6FDKRWMNNSWR7UCGFNFCX7OIG3XH4Q, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.207190219 (915h24m2.078273254s) [noise=true , address=45.91.116.9:4262 , piece_id=LNLGQ3ZKC4FSXMLLUGKM7RMDVR5BIDLUALFH25QQV2TCFHK6VFQA, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.235451699 (915h24m2.106534744s) [noise=true , address=45.91.116.6:4973 , piece_id=O7HE3VVAJRQAO4B52YR3YQ5OBGMZAKDRFHYLVKTYI46QAT5NVRYQ, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.264976344 (915h24m2.136059389s) [noise=true , address=145.79.227.11:4731 , piece_id=LYMJBPG2ZPATR3UNXLLDC5SY22EOVC6YBCEQIEE3XGZ4RJRSJXZQ, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.278399723 (915h24m2.149482768s) [noise=true , address=145.79.227.45:4231 , piece_id=O36XJLX6OBD4YFYLNNWSXCJI3K5WZMLFNDPYASYSGSGWY6DJSWRQ, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.278424661 (915h24m2.149507696s) [seg_pos={0 0}, upload=801199]: Segment reached optimal threshold of 13 pieces. Jan 12 18:53:02.278758596 (915h24m2.149841671s) [noise=true , address=45.91.116.5:4300 , piece_id=ABVDLNQ7ZD7M3RVJLCZNKAHWU5MVJTKVLHJLERRXCB4ZVZHELSCA, seg_pos={0 0}, upload=801199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 12 18:53:02.278800566 (915h24m2.149883621s) [noise=true , address=45.91.116.5:4300 , piece_id=ABVDLNQ7ZD7M3RVJLCZNKAHWU5MVJTKVLHJLERRXCB4ZVZHELSCA, seg_pos={0 0}, upload=801199]: Long tail cancellation due to optimal threshold Jan 12 18:53:02.278925184 (915h24m2.150008229s) [noise=true , address=145.79.227.20:4439 , piece_id=R76OUH3NPZFJMHP7C5YLH44RLX2J7VIC4R3EHUZFTMCK7BAKPNJQ, seg_pos={0 0}, upload=801199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn): protocol: expected piece hash; context canceled; EOF Jan 12 18:53:02.278936645 (915h24m2.1500197s) [noise=true , address=145.79.227.20:4439 , piece_id=R76OUH3NPZFJMHP7C5YLH44RLX2J7VIC4R3EHUZFTMCK7BAKPNJQ, seg_pos={0 0}, upload=801199]: Long tail cancellation due to optimal threshold Jan 12 18:53:02.279000747 (915h24m2.150083792s) [noise=true , address=45.91.116.8:4047 , piece_id=TYFKNAOCLKSQBEUOX7JCUAVECLUSOJMZL3BQNWOX7QK7WRRIEI6Q, seg_pos={0 0}, upload=801199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:56336->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:56336->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56336->45.91.116.8:4047: use of closed network connection Jan 12 18:53:02.279008672 (915h24m2.150091717s) [noise=true , address=45.91.116.8:4047 , piece_id=TYFKNAOCLKSQBEUOX7JCUAVECLUSOJMZL3BQNWOX7QK7WRRIEI6Q, seg_pos={0 0}, upload=801199]: Long tail cancellation due to optimal threshold Jan 12 18:53:02.279037688 (915h24m2.150120723s) [seg_pos={0 0}, upload=801199]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 18:53:02.279149039 (915h24m2.150232115s) [upload=801199]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 18:53:02.279156413 (915h24m2.150239468s) [upload=801199]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 18:53:02.601181307 (915h24m2.472264352s) [upload=801199]: Done uploading segments... Jan 12 18:53:02.601202908 (915h24m2.472285953s) [upload=801199]: Upload finished. err: <nil> Duration: 5.81170509s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 28 out of 60 is failed (threshold=6) [XXXX.X.XXX.X.XXXXX.XX....X.XXX.X..X.X.X....XX...........XX..] duration: 3.551252241s trace: https://jaeger.dev.storj.io/trace/70bcd4df5d87a249 FAIL (2026-01-12 18:54:27 UTC) orbiter warning: context deadline exceeded Duration: 3.551276837s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X.............X..] duration: 5.916238192s trace: https://jaeger.dev.storj.io/trace/512bcb9bf08d2da9 FAIL (2026-01-12 18:54:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 12 18:52:59.379576882 (915h23m59.250659917s) [upload=801199]: Starting upload Jan 12 18:52:59.379630744 (915h23m59.250713779s) [upload=801199]: Done starting upload Jan 12 18:52:59.379719844 (915h23m59.250802879s) [upload=801199]: Uploading segments... Jan 12 18:52:59.379879347 (915h23m59.250962382s) [upload=801199]: Got next segment. Inline: false Jan 12 18:52:59.379917681 (915h23m59.251000726s) [upload=801199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 18:52:59.379922670 (915h23m59.251005716s) [upload=801199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 18:52:59.945729741 (915h23m59.816812776s) [seg_pos={0 0}, upload=801199]: Begin upload segment... Jan 12 18:52:59.945777812 (915h23m59.816860847s) [seg_pos={0 0}, upload=801199]: Done begin upload segment. Jan 12 18:52:59.945872483 (915h23m59.816955518s) [noise=true , address=145.79.227.20:4439 , piece_id=R76OUH3NPZFJMHP7C5YLH44RLX2J7VIC4R3EHUZFTMCK7BAKPNJQ, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.945889696 (915h23m59.816972731s) [noise=true , address=77.90.31.21:4880 , piece_id=IIHNAKYPKV3DIUNZOCGK3ZVOYQXFYWEKMCWTULKYM76J2ZXRC53A, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.945974337 (915h23m59.817057372s) [noise=true , address=45.91.116.9:4262 , piece_id=LNLGQ3ZKC4FSXMLLUGKM7RMDVR5BIDLUALFH25QQV2TCFHK6VFQA, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.946516238 (915h23m59.817599273s) [noise=true , address=77.90.31.12:4590 , piece_id=3EW5KXCEXD37TTKUAMO3OJEQEZGYGCVDRWCKUC7P7KPGMJUVCOVA, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.946888608 (915h23m59.817971643s) [noise=true , address=77.90.31.11:4555 , piece_id=DFQXR5X3OCZIVCZSESGUL3GK2F6E7EKVHMOOI5VDOM452ERKIN6Q, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.946895791 (915h23m59.817978826s) [noise=true , address=77.90.31.18:4808 , piece_id=R2VL26Y4B2C2IVLT4OZ5F3U2HGIIO75IIEO24QUAGSNQRCA7VC3A, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.947217193 (915h23m59.818300228s) [noise=true , address=45.91.116.6:4973 , piece_id=O7HE3VVAJRQAO4B52YR3YQ5OBGMZAKDRFHYLVKTYI46QAT5NVRYQ, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.947275544 (915h23m59.818358579s) [noise=true , address=145.79.227.7:4295 , piece_id=NK4344JKJ6PFU6T2NY62MQBYILLI52ITNI2FDO36MRY64CCZEQTA, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.947584833 (915h23m59.818667868s) [noise=true , address=145.79.227.45:4231 , piece_id=O36XJLX6OBD4YFYLNNWSXCJI3K5WZMLFNDPYASYSGSGWY6DJSWRQ, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.947599621 (915h23m59.818682656s) [noise=true , address=77.90.31.10:4519 , piece_id=2PNRYI7FPX7UMRI4FAQWA3TMITJ5SESOPKJF5U4Q6WEMPACR6NRQ, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.947961120 (915h23m59.819044155s) [noise=true , address=45.91.116.12:4227 , piece_id=OFOCOYEFOAZGKJOFWVNLYM6FDKRWMNNSWR7UCGFNFCX7OIG3XH4Q, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.948035271 (915h23m59.819118316s) [noise=true , address=145.79.227.11:4731 , piece_id=LYMJBPG2ZPATR3UNXLLDC5SY22EOVC6YBCEQIEE3XGZ4RJRSJXZQ, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.948293923 (915h23m59.819376958s) [noise=true , address=45.91.116.8:4047 , piece_id=TYFKNAOCLKSQBEUOX7JCUAVECLUSOJMZL3BQNWOX7QK7WRRIEI6Q, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.948360670 (915h23m59.819443705s) [noise=true , address=145.79.227.52:4446 , piece_id=2JRS3WPJ2PAFE3B3EUZO27DQS4HDKPHXZYIGIWFJTDWRGSV3IRIA, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.948598854 (915h23m59.819681889s) [noise=true , address=145.79.227.16:4910 , piece_id=I2O3LBJLA3NUICDDB52PRRQWXZWE7VE2F73C5TZ6G5CPQ55EI47A, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.948693183 (915h23m59.819776218s) [noise=true , address=45.91.116.5:4300 , piece_id=ABVDLNQ7ZD7M3RVJLCZNKAHWU5MVJTKVLHJLERRXCB4ZVZHELSCA, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:53:01.738921723 (915h24m1.610004768s) [upload=801199]: Next returned nil segment Jan 12 18:53:01.738939648 (915h24m1.610022693s) [upload=801199]: Waiting for error group managing segments... Jan 12 18:53:02.073911963 (915h24m1.944995008s) [noise=true , address=77.90.31.10:4519 , piece_id=2PNRYI7FPX7UMRI4FAQWA3TMITJ5SESOPKJF5U4Q6WEMPACR6NRQ, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.081381215 (915h24m1.95246426s) [noise=true , address=145.79.227.16:4910 , piece_id=I2O3LBJLA3NUICDDB52PRRQWXZWE7VE2F73C5TZ6G5CPQ55EI47A, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.089752244 (915h24m1.960835289s) [noise=true , address=77.90.31.11:4555 , piece_id=DFQXR5X3OCZIVCZSESGUL3GK2F6E7EKVHMOOI5VDOM452ERKIN6Q, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.099125060 (915h24m1.970208105s) [noise=true , address=77.90.31.12:4590 , piece_id=3EW5KXCEXD37TTKUAMO3OJEQEZGYGCVDRWCKUC7P7KPGMJUVCOVA, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.119372481 (915h24m1.990455526s) [noise=true , address=77.90.31.21:4880 , piece_id=IIHNAKYPKV3DIUNZOCGK3ZVOYQXFYWEKMCWTULKYM76J2ZXRC53A, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.160786966 (915h24m2.031870011s) [noise=true , address=145.79.227.7:4295 , piece_id=NK4344JKJ6PFU6T2NY62MQBYILLI52ITNI2FDO36MRY64CCZEQTA, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.173398348 (915h24m2.044481383s) [noise=true , address=77.90.31.18:4808 , piece_id=R2VL26Y4B2C2IVLT4OZ5F3U2HGIIO75IIEO24QUAGSNQRCA7VC3A, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.191338316 (915h24m2.062421361s) [noise=true , address=145.79.227.52:4446 , piece_id=2JRS3WPJ2PAFE3B3EUZO27DQS4HDKPHXZYIGIWFJTDWRGSV3IRIA, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.193169993 (915h24m2.064253038s) [noise=true , address=45.91.116.12:4227 , piece_id=OFOCOYEFOAZGKJOFWVNLYM6FDKRWMNNSWR7UCGFNFCX7OIG3XH4Q, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.207190219 (915h24m2.078273254s) [noise=true , address=45.91.116.9:4262 , piece_id=LNLGQ3ZKC4FSXMLLUGKM7RMDVR5BIDLUALFH25QQV2TCFHK6VFQA, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.235451699 (915h24m2.106534744s) [noise=true , address=45.91.116.6:4973 , piece_id=O7HE3VVAJRQAO4B52YR3YQ5OBGMZAKDRFHYLVKTYI46QAT5NVRYQ, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.264976344 (915h24m2.136059389s) [noise=true , address=145.79.227.11:4731 , piece_id=LYMJBPG2ZPATR3UNXLLDC5SY22EOVC6YBCEQIEE3XGZ4RJRSJXZQ, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.278399723 (915h24m2.149482768s) [noise=true , address=145.79.227.45:4231 , piece_id=O36XJLX6OBD4YFYLNNWSXCJI3K5WZMLFNDPYASYSGSGWY6DJSWRQ, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.278424661 (915h24m2.149507696s) [seg_pos={0 0}, upload=801199]: Segment reached optimal threshold of 13 pieces. Jan 12 18:53:02.278758596 (915h24m2.149841671s) [noise=true , address=45.91.116.5:4300 , piece_id=ABVDLNQ7ZD7M3RVJLCZNKAHWU5MVJTKVLHJLERRXCB4ZVZHELSCA, seg_pos={0 0}, upload=801199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 12 18:53:02.278800566 (915h24m2.149883621s) [noise=true , address=45.91.116.5:4300 , piece_id=ABVDLNQ7ZD7M3RVJLCZNKAHWU5MVJTKVLHJLERRXCB4ZVZHELSCA, seg_pos={0 0}, upload=801199]: Long tail cancellation due to optimal threshold Jan 12 18:53:02.278925184 (915h24m2.150008229s) [noise=true , address=145.79.227.20:4439 , piece_id=R76OUH3NPZFJMHP7C5YLH44RLX2J7VIC4R3EHUZFTMCK7BAKPNJQ, seg_pos={0 0}, upload=801199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn): protocol: expected piece hash; context canceled; EOF Jan 12 18:53:02.278936645 (915h24m2.1500197s) [noise=true , address=145.79.227.20:4439 , piece_id=R76OUH3NPZFJMHP7C5YLH44RLX2J7VIC4R3EHUZFTMCK7BAKPNJQ, seg_pos={0 0}, upload=801199]: Long tail cancellation due to optimal threshold Jan 12 18:53:02.279000747 (915h24m2.150083792s) [noise=true , address=45.91.116.8:4047 , piece_id=TYFKNAOCLKSQBEUOX7JCUAVECLUSOJMZL3BQNWOX7QK7WRRIEI6Q, seg_pos={0 0}, upload=801199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:56336->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:56336->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56336->45.91.116.8:4047: use of closed network connection Jan 12 18:53:02.279008672 (915h24m2.150091717s) [noise=true , address=45.91.116.8:4047 , piece_id=TYFKNAOCLKSQBEUOX7JCUAVECLUSOJMZL3BQNWOX7QK7WRRIEI6Q, seg_pos={0 0}, upload=801199]: Long tail cancellation due to optimal threshold Jan 12 18:53:02.279037688 (915h24m2.150120723s) [seg_pos={0 0}, upload=801199]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 18:53:02.279149039 (915h24m2.150232115s) [upload=801199]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 18:53:02.279156413 (915h24m2.150239468s) [upload=801199]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 18:53:02.601181307 (915h24m2.472264352s) [upload=801199]: Done uploading segments... Jan 12 18:53:02.601202908 (915h24m2.472285953s) [upload=801199]: Upload finished. err: <nil> Duration: 5.916253051s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [XXXXX.X.XXX.X.XXXXX.XX....X.XXX.X..X.X.X....XX...........XX.] duration: 3.585767567s trace: https://jaeger.dev.storj.io/trace/70bcd4df5d87a249 FAIL (2026-01-12 18:54:12 UTC) orbiter warning: context deadline exceeded Duration: 3.585793568s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X.............X.] duration: 16.375120858s trace: https://jaeger.dev.storj.io/trace/512bcb9bf08d2da9 FAIL (2026-01-12 18:53:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 12 18:52:59.379576882 (915h23m59.250659917s) [upload=801199]: Starting upload Jan 12 18:52:59.379630744 (915h23m59.250713779s) [upload=801199]: Done starting upload Jan 12 18:52:59.379719844 (915h23m59.250802879s) [upload=801199]: Uploading segments... Jan 12 18:52:59.379879347 (915h23m59.250962382s) [upload=801199]: Got next segment. Inline: false Jan 12 18:52:59.379917681 (915h23m59.251000726s) [upload=801199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 18:52:59.379922670 (915h23m59.251005716s) [upload=801199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 18:52:59.945729741 (915h23m59.816812776s) [seg_pos={0 0}, upload=801199]: Begin upload segment... Jan 12 18:52:59.945777812 (915h23m59.816860847s) [seg_pos={0 0}, upload=801199]: Done begin upload segment. Jan 12 18:52:59.945872483 (915h23m59.816955518s) [noise=true , address=145.79.227.20:4439 , piece_id=R76OUH3NPZFJMHP7C5YLH44RLX2J7VIC4R3EHUZFTMCK7BAKPNJQ, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.945889696 (915h23m59.816972731s) [noise=true , address=77.90.31.21:4880 , piece_id=IIHNAKYPKV3DIUNZOCGK3ZVOYQXFYWEKMCWTULKYM76J2ZXRC53A, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.945974337 (915h23m59.817057372s) [noise=true , address=45.91.116.9:4262 , piece_id=LNLGQ3ZKC4FSXMLLUGKM7RMDVR5BIDLUALFH25QQV2TCFHK6VFQA, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.946516238 (915h23m59.817599273s) [noise=true , address=77.90.31.12:4590 , piece_id=3EW5KXCEXD37TTKUAMO3OJEQEZGYGCVDRWCKUC7P7KPGMJUVCOVA, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.946888608 (915h23m59.817971643s) [noise=true , address=77.90.31.11:4555 , piece_id=DFQXR5X3OCZIVCZSESGUL3GK2F6E7EKVHMOOI5VDOM452ERKIN6Q, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.946895791 (915h23m59.817978826s) [noise=true , address=77.90.31.18:4808 , piece_id=R2VL26Y4B2C2IVLT4OZ5F3U2HGIIO75IIEO24QUAGSNQRCA7VC3A, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.947217193 (915h23m59.818300228s) [noise=true , address=45.91.116.6:4973 , piece_id=O7HE3VVAJRQAO4B52YR3YQ5OBGMZAKDRFHYLVKTYI46QAT5NVRYQ, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.947275544 (915h23m59.818358579s) [noise=true , address=145.79.227.7:4295 , piece_id=NK4344JKJ6PFU6T2NY62MQBYILLI52ITNI2FDO36MRY64CCZEQTA, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.947584833 (915h23m59.818667868s) [noise=true , address=145.79.227.45:4231 , piece_id=O36XJLX6OBD4YFYLNNWSXCJI3K5WZMLFNDPYASYSGSGWY6DJSWRQ, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.947599621 (915h23m59.818682656s) [noise=true , address=77.90.31.10:4519 , piece_id=2PNRYI7FPX7UMRI4FAQWA3TMITJ5SESOPKJF5U4Q6WEMPACR6NRQ, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.947961120 (915h23m59.819044155s) [noise=true , address=45.91.116.12:4227 , piece_id=OFOCOYEFOAZGKJOFWVNLYM6FDKRWMNNSWR7UCGFNFCX7OIG3XH4Q, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.948035271 (915h23m59.819118316s) [noise=true , address=145.79.227.11:4731 , piece_id=LYMJBPG2ZPATR3UNXLLDC5SY22EOVC6YBCEQIEE3XGZ4RJRSJXZQ, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.948293923 (915h23m59.819376958s) [noise=true , address=45.91.116.8:4047 , piece_id=TYFKNAOCLKSQBEUOX7JCUAVECLUSOJMZL3BQNWOX7QK7WRRIEI6Q, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.948360670 (915h23m59.819443705s) [noise=true , address=145.79.227.52:4446 , piece_id=2JRS3WPJ2PAFE3B3EUZO27DQS4HDKPHXZYIGIWFJTDWRGSV3IRIA, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.948598854 (915h23m59.819681889s) [noise=true , address=145.79.227.16:4910 , piece_id=I2O3LBJLA3NUICDDB52PRRQWXZWE7VE2F73C5TZ6G5CPQ55EI47A, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.948693183 (915h23m59.819776218s) [noise=true , address=45.91.116.5:4300 , piece_id=ABVDLNQ7ZD7M3RVJLCZNKAHWU5MVJTKVLHJLERRXCB4ZVZHELSCA, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:53:01.738921723 (915h24m1.610004768s) [upload=801199]: Next returned nil segment Jan 12 18:53:01.738939648 (915h24m1.610022693s) [upload=801199]: Waiting for error group managing segments... Jan 12 18:53:02.073911963 (915h24m1.944995008s) [noise=true , address=77.90.31.10:4519 , piece_id=2PNRYI7FPX7UMRI4FAQWA3TMITJ5SESOPKJF5U4Q6WEMPACR6NRQ, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.081381215 (915h24m1.95246426s) [noise=true , address=145.79.227.16:4910 , piece_id=I2O3LBJLA3NUICDDB52PRRQWXZWE7VE2F73C5TZ6G5CPQ55EI47A, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.089752244 (915h24m1.960835289s) [noise=true , address=77.90.31.11:4555 , piece_id=DFQXR5X3OCZIVCZSESGUL3GK2F6E7EKVHMOOI5VDOM452ERKIN6Q, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.099125060 (915h24m1.970208105s) [noise=true , address=77.90.31.12:4590 , piece_id=3EW5KXCEXD37TTKUAMO3OJEQEZGYGCVDRWCKUC7P7KPGMJUVCOVA, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.119372481 (915h24m1.990455526s) [noise=true , address=77.90.31.21:4880 , piece_id=IIHNAKYPKV3DIUNZOCGK3ZVOYQXFYWEKMCWTULKYM76J2ZXRC53A, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.160786966 (915h24m2.031870011s) [noise=true , address=145.79.227.7:4295 , piece_id=NK4344JKJ6PFU6T2NY62MQBYILLI52ITNI2FDO36MRY64CCZEQTA, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.173398348 (915h24m2.044481383s) [noise=true , address=77.90.31.18:4808 , piece_id=R2VL26Y4B2C2IVLT4OZ5F3U2HGIIO75IIEO24QUAGSNQRCA7VC3A, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.191338316 (915h24m2.062421361s) [noise=true , address=145.79.227.52:4446 , piece_id=2JRS3WPJ2PAFE3B3EUZO27DQS4HDKPHXZYIGIWFJTDWRGSV3IRIA, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.193169993 (915h24m2.064253038s) [noise=true , address=45.91.116.12:4227 , piece_id=OFOCOYEFOAZGKJOFWVNLYM6FDKRWMNNSWR7UCGFNFCX7OIG3XH4Q, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.207190219 (915h24m2.078273254s) [noise=true , address=45.91.116.9:4262 , piece_id=LNLGQ3ZKC4FSXMLLUGKM7RMDVR5BIDLUALFH25QQV2TCFHK6VFQA, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.235451699 (915h24m2.106534744s) [noise=true , address=45.91.116.6:4973 , piece_id=O7HE3VVAJRQAO4B52YR3YQ5OBGMZAKDRFHYLVKTYI46QAT5NVRYQ, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.264976344 (915h24m2.136059389s) [noise=true , address=145.79.227.11:4731 , piece_id=LYMJBPG2ZPATR3UNXLLDC5SY22EOVC6YBCEQIEE3XGZ4RJRSJXZQ, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.278399723 (915h24m2.149482768s) [noise=true , address=145.79.227.45:4231 , piece_id=O36XJLX6OBD4YFYLNNWSXCJI3K5WZMLFNDPYASYSGSGWY6DJSWRQ, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.278424661 (915h24m2.149507696s) [seg_pos={0 0}, upload=801199]: Segment reached optimal threshold of 13 pieces. Jan 12 18:53:02.278758596 (915h24m2.149841671s) [noise=true , address=45.91.116.5:4300 , piece_id=ABVDLNQ7ZD7M3RVJLCZNKAHWU5MVJTKVLHJLERRXCB4ZVZHELSCA, seg_pos={0 0}, upload=801199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 12 18:53:02.278800566 (915h24m2.149883621s) [noise=true , address=45.91.116.5:4300 , piece_id=ABVDLNQ7ZD7M3RVJLCZNKAHWU5MVJTKVLHJLERRXCB4ZVZHELSCA, seg_pos={0 0}, upload=801199]: Long tail cancellation due to optimal threshold Jan 12 18:53:02.278925184 (915h24m2.150008229s) [noise=true , address=145.79.227.20:4439 , piece_id=R76OUH3NPZFJMHP7C5YLH44RLX2J7VIC4R3EHUZFTMCK7BAKPNJQ, seg_pos={0 0}, upload=801199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn): protocol: expected piece hash; context canceled; EOF Jan 12 18:53:02.278936645 (915h24m2.1500197s) [noise=true , address=145.79.227.20:4439 , piece_id=R76OUH3NPZFJMHP7C5YLH44RLX2J7VIC4R3EHUZFTMCK7BAKPNJQ, seg_pos={0 0}, upload=801199]: Long tail cancellation due to optimal threshold Jan 12 18:53:02.279000747 (915h24m2.150083792s) [noise=true , address=45.91.116.8:4047 , piece_id=TYFKNAOCLKSQBEUOX7JCUAVECLUSOJMZL3BQNWOX7QK7WRRIEI6Q, seg_pos={0 0}, upload=801199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:56336->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:56336->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56336->45.91.116.8:4047: use of closed network connection Jan 12 18:53:02.279008672 (915h24m2.150091717s) [noise=true , address=45.91.116.8:4047 , piece_id=TYFKNAOCLKSQBEUOX7JCUAVECLUSOJMZL3BQNWOX7QK7WRRIEI6Q, seg_pos={0 0}, upload=801199]: Long tail cancellation due to optimal threshold Jan 12 18:53:02.279037688 (915h24m2.150120723s) [seg_pos={0 0}, upload=801199]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 18:53:02.279149039 (915h24m2.150232115s) [upload=801199]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 18:53:02.279156413 (915h24m2.150239468s) [upload=801199]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 18:53:02.601181307 (915h24m2.472264352s) [upload=801199]: Done uploading segments... Jan 12 18:53:02.601202908 (915h24m2.472285953s) [upload=801199]: Upload finished. err: <nil> Duration: 16.375137549s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [XXXXXX.X.XXX.X.XXXXX.XX....X.XXX.X..X.X.X....XX...........XX] duration: 5.001811075s trace: https://jaeger.dev.storj.io/trace/70bcd4df5d87a249 FAIL (2026-01-12 18:53:29 UTC) orbiter warning: context deadline exceeded Duration: 5.001826775s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X.............X] duration: 30.002683859s trace: https://jaeger.dev.storj.io/trace/512bcb9bf08d2da9 FAIL (2026-01-12 18:53:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Jan 12 18:52:59.379576882 (915h23m59.250659917s) [upload=801199]: Starting upload Jan 12 18:52:59.379630744 (915h23m59.250713779s) [upload=801199]: Done starting upload Jan 12 18:52:59.379719844 (915h23m59.250802879s) [upload=801199]: Uploading segments... Jan 12 18:52:59.379879347 (915h23m59.250962382s) [upload=801199]: Got next segment. Inline: false Jan 12 18:52:59.379917681 (915h23m59.251000726s) [upload=801199]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 18:52:59.379922670 (915h23m59.251005716s) [upload=801199]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 18:52:59.945729741 (915h23m59.816812776s) [seg_pos={0 0}, upload=801199]: Begin upload segment... Jan 12 18:52:59.945777812 (915h23m59.816860847s) [seg_pos={0 0}, upload=801199]: Done begin upload segment. Jan 12 18:52:59.945872483 (915h23m59.816955518s) [noise=true , address=145.79.227.20:4439 , piece_id=R76OUH3NPZFJMHP7C5YLH44RLX2J7VIC4R3EHUZFTMCK7BAKPNJQ, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.945889696 (915h23m59.816972731s) [noise=true , address=77.90.31.21:4880 , piece_id=IIHNAKYPKV3DIUNZOCGK3ZVOYQXFYWEKMCWTULKYM76J2ZXRC53A, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.945974337 (915h23m59.817057372s) [noise=true , address=45.91.116.9:4262 , piece_id=LNLGQ3ZKC4FSXMLLUGKM7RMDVR5BIDLUALFH25QQV2TCFHK6VFQA, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.946516238 (915h23m59.817599273s) [noise=true , address=77.90.31.12:4590 , piece_id=3EW5KXCEXD37TTKUAMO3OJEQEZGYGCVDRWCKUC7P7KPGMJUVCOVA, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.946888608 (915h23m59.817971643s) [noise=true , address=77.90.31.11:4555 , piece_id=DFQXR5X3OCZIVCZSESGUL3GK2F6E7EKVHMOOI5VDOM452ERKIN6Q, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.946895791 (915h23m59.817978826s) [noise=true , address=77.90.31.18:4808 , piece_id=R2VL26Y4B2C2IVLT4OZ5F3U2HGIIO75IIEO24QUAGSNQRCA7VC3A, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.947217193 (915h23m59.818300228s) [noise=true , address=45.91.116.6:4973 , piece_id=O7HE3VVAJRQAO4B52YR3YQ5OBGMZAKDRFHYLVKTYI46QAT5NVRYQ, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.947275544 (915h23m59.818358579s) [noise=true , address=145.79.227.7:4295 , piece_id=NK4344JKJ6PFU6T2NY62MQBYILLI52ITNI2FDO36MRY64CCZEQTA, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.947584833 (915h23m59.818667868s) [noise=true , address=145.79.227.45:4231 , piece_id=O36XJLX6OBD4YFYLNNWSXCJI3K5WZMLFNDPYASYSGSGWY6DJSWRQ, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.947599621 (915h23m59.818682656s) [noise=true , address=77.90.31.10:4519 , piece_id=2PNRYI7FPX7UMRI4FAQWA3TMITJ5SESOPKJF5U4Q6WEMPACR6NRQ, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.947961120 (915h23m59.819044155s) [noise=true , address=45.91.116.12:4227 , piece_id=OFOCOYEFOAZGKJOFWVNLYM6FDKRWMNNSWR7UCGFNFCX7OIG3XH4Q, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.948035271 (915h23m59.819118316s) [noise=true , address=145.79.227.11:4731 , piece_id=LYMJBPG2ZPATR3UNXLLDC5SY22EOVC6YBCEQIEE3XGZ4RJRSJXZQ, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.948293923 (915h23m59.819376958s) [noise=true , address=45.91.116.8:4047 , piece_id=TYFKNAOCLKSQBEUOX7JCUAVECLUSOJMZL3BQNWOX7QK7WRRIEI6Q, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.948360670 (915h23m59.819443705s) [noise=true , address=145.79.227.52:4446 , piece_id=2JRS3WPJ2PAFE3B3EUZO27DQS4HDKPHXZYIGIWFJTDWRGSV3IRIA, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.948598854 (915h23m59.819681889s) [noise=true , address=145.79.227.16:4910 , piece_id=I2O3LBJLA3NUICDDB52PRRQWXZWE7VE2F73C5TZ6G5CPQ55EI47A, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:52:59.948693183 (915h23m59.819776218s) [noise=true , address=45.91.116.5:4300 , piece_id=ABVDLNQ7ZD7M3RVJLCZNKAHWU5MVJTKVLHJLERRXCB4ZVZHELSCA, seg_pos={0 0}, upload=801199]: Uploading piece... Jan 12 18:53:01.738921723 (915h24m1.610004768s) [upload=801199]: Next returned nil segment Jan 12 18:53:01.738939648 (915h24m1.610022693s) [upload=801199]: Waiting for error group managing segments... Jan 12 18:53:02.073911963 (915h24m1.944995008s) [noise=true , address=77.90.31.10:4519 , piece_id=2PNRYI7FPX7UMRI4FAQWA3TMITJ5SESOPKJF5U4Q6WEMPACR6NRQ, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.081381215 (915h24m1.95246426s) [noise=true , address=145.79.227.16:4910 , piece_id=I2O3LBJLA3NUICDDB52PRRQWXZWE7VE2F73C5TZ6G5CPQ55EI47A, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.089752244 (915h24m1.960835289s) [noise=true , address=77.90.31.11:4555 , piece_id=DFQXR5X3OCZIVCZSESGUL3GK2F6E7EKVHMOOI5VDOM452ERKIN6Q, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.099125060 (915h24m1.970208105s) [noise=true , address=77.90.31.12:4590 , piece_id=3EW5KXCEXD37TTKUAMO3OJEQEZGYGCVDRWCKUC7P7KPGMJUVCOVA, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.119372481 (915h24m1.990455526s) [noise=true , address=77.90.31.21:4880 , piece_id=IIHNAKYPKV3DIUNZOCGK3ZVOYQXFYWEKMCWTULKYM76J2ZXRC53A, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.160786966 (915h24m2.031870011s) [noise=true , address=145.79.227.7:4295 , piece_id=NK4344JKJ6PFU6T2NY62MQBYILLI52ITNI2FDO36MRY64CCZEQTA, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.173398348 (915h24m2.044481383s) [noise=true , address=77.90.31.18:4808 , piece_id=R2VL26Y4B2C2IVLT4OZ5F3U2HGIIO75IIEO24QUAGSNQRCA7VC3A, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.191338316 (915h24m2.062421361s) [noise=true , address=145.79.227.52:4446 , piece_id=2JRS3WPJ2PAFE3B3EUZO27DQS4HDKPHXZYIGIWFJTDWRGSV3IRIA, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.193169993 (915h24m2.064253038s) [noise=true , address=45.91.116.12:4227 , piece_id=OFOCOYEFOAZGKJOFWVNLYM6FDKRWMNNSWR7UCGFNFCX7OIG3XH4Q, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.207190219 (915h24m2.078273254s) [noise=true , address=45.91.116.9:4262 , piece_id=LNLGQ3ZKC4FSXMLLUGKM7RMDVR5BIDLUALFH25QQV2TCFHK6VFQA, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.235451699 (915h24m2.106534744s) [noise=true , address=45.91.116.6:4973 , piece_id=O7HE3VVAJRQAO4B52YR3YQ5OBGMZAKDRFHYLVKTYI46QAT5NVRYQ, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.264976344 (915h24m2.136059389s) [noise=true , address=145.79.227.11:4731 , piece_id=LYMJBPG2ZPATR3UNXLLDC5SY22EOVC6YBCEQIEE3XGZ4RJRSJXZQ, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.278399723 (915h24m2.149482768s) [noise=true , address=145.79.227.45:4231 , piece_id=O36XJLX6OBD4YFYLNNWSXCJI3K5WZMLFNDPYASYSGSGWY6DJSWRQ, seg_pos={0 0}, upload=801199]: Done uploading piece. err: <nil> Jan 12 18:53:02.278424661 (915h24m2.149507696s) [seg_pos={0 0}, upload=801199]: Segment reached optimal threshold of 13 pieces. Jan 12 18:53:02.278758596 (915h24m2.149841671s) [noise=true , address=45.91.116.5:4300 , piece_id=ABVDLNQ7ZD7M3RVJLCZNKAHWU5MVJTKVLHJLERRXCB4ZVZHELSCA, seg_pos={0 0}, upload=801199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 12 18:53:02.278800566 (915h24m2.149883621s) [noise=true , address=45.91.116.5:4300 , piece_id=ABVDLNQ7ZD7M3RVJLCZNKAHWU5MVJTKVLHJLERRXCB4ZVZHELSCA, seg_pos={0 0}, upload=801199]: Long tail cancellation due to optimal threshold Jan 12 18:53:02.278925184 (915h24m2.150008229s) [noise=true , address=145.79.227.20:4439 , piece_id=R76OUH3NPZFJMHP7C5YLH44RLX2J7VIC4R3EHUZFTMCK7BAKPNJQ, seg_pos={0 0}, upload=801199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1wgkbc3Mkej7supxhYMtUdBvn9t45i6kmAokAUkqFxEpehMjAn): protocol: expected piece hash; context canceled; EOF Jan 12 18:53:02.278936645 (915h24m2.1500197s) [noise=true , address=145.79.227.20:4439 , piece_id=R76OUH3NPZFJMHP7C5YLH44RLX2J7VIC4R3EHUZFTMCK7BAKPNJQ, seg_pos={0 0}, upload=801199]: Long tail cancellation due to optimal threshold Jan 12 18:53:02.279000747 (915h24m2.150083792s) [noise=true , address=45.91.116.8:4047 , piece_id=TYFKNAOCLKSQBEUOX7JCUAVECLUSOJMZL3BQNWOX7QK7WRRIEI6Q, seg_pos={0 0}, upload=801199]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:56336->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:56336->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56336->45.91.116.8:4047: use of closed network connection Jan 12 18:53:02.279008672 (915h24m2.150091717s) [noise=true , address=45.91.116.8:4047 , piece_id=TYFKNAOCLKSQBEUOX7JCUAVECLUSOJMZL3BQNWOX7QK7WRRIEI6Q, seg_pos={0 0}, upload=801199]: Long tail cancellation due to optimal threshold Jan 12 18:53:02.279037688 (915h24m2.150120723s) [seg_pos={0 0}, upload=801199]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 18:53:02.279149039 (915h24m2.150232115s) [upload=801199]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 18:53:02.279156413 (915h24m2.150239468s) [upload=801199]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 18:53:02.601181307 (915h24m2.472264352s) [upload=801199]: Done uploading segments... Jan 12 18:53:02.601202908 (915h24m2.472285953s) [upload=801199]: Upload finished. err: <nil> Duration: 30.00270493s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [.XXXXXX.X.XXX.X.XXXXX.XX....X.XXX.X..X.X.X....XX...........X] duration: 5.001604511s trace: https://jaeger.dev.storj.io/trace/1a0f6fd2fb02cce4 FAIL (2026-01-12 18:53:07 UTC) orbiter warning: context deadline exceeded Duration: 5.001623937s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [X.XXXXXX.X.XXX.X.XXXXX.XX....X.XXX.X..X.X.X....XX...........] duration: 3.745580838s trace: https://jaeger.dev.storj.io/trace/64e8accbe0a8a091 FAIL (2026-01-12 18:52:20 UTC) orbiter warning: context deadline exceeded Duration: 3.745599535s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.825293393s trace: https://jaeger.dev.storj.io/trace/de69fa036d8b02 FAIL (2026-01-12 18:51:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ) context deadline exceeded; piecestore: (Node ID: 1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE, Piece ID: EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ) context deadline exceeded; piecestore: (Node ID: 18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29, Piece ID: HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ) context deadline exceeded ======LOGS====== Jan 12 18:43:13.858719809 (915h14m13.729802844s) [upload=801054]: Starting upload Jan 12 18:43:13.858805642 (915h14m13.729888677s) [upload=801054]: Done starting upload Jan 12 18:43:13.858880885 (915h14m13.72996393s) [upload=801054]: Uploading segments... Jan 12 18:43:13.859066038 (915h14m13.730149083s) [upload=801054]: Got next segment. Inline: false Jan 12 18:43:13.859125010 (915h14m13.730208045s) [upload=801054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 18:43:13.859131923 (915h14m13.730214968s) [upload=801054]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 18:43:15.358668982 (915h14m15.229752017s) [seg_pos={0 0}, upload=801054]: Begin upload segment... Jan 12 18:43:15.358714368 (915h14m15.229797413s) [seg_pos={0 0}, upload=801054]: Done begin upload segment. Jan 12 18:43:15.358745317 (915h14m15.229828352s) [noise=true , address=77.90.31.12:4590 , piece_id=LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.358908408 (915h14m15.229991453s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.359387038 (915h14m15.230470083s) [noise=true , address=145.79.227.44:4194 , piece_id=PKZFIIDQYD5O2P25BKIMA45XLXVXZQSLV5H7A2JJGG7XOAHOO2QA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.359851062 (915h14m15.230934107s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360286561 (915h14m15.231369606s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360600609 (915h14m15.231683654s) [noise=true , address=145.79.227.29:4943 , piece_id=EHLLIAZBTJOBDU2VLA2JXDVQK67WSCRU3FR24HBWK2QF2A5XUERA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360877186 (915h14m15.231960231s) [noise=true , address=77.90.31.19:4841 , piece_id=PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.361298538 (915h14m15.232381573s) [noise=true , address=145.79.227.49:4337 , piece_id=EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.362993415 (915h14m15.23407645s) [noise=true , address=45.91.116.14:4189 , piece_id=I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.363095780 (915h14m15.234178825s) [noise=true , address=45.91.116.7:4012 , piece_id=YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.364580467 (915h14m15.235663502s) [noise=true , address=145.79.227.16:4909 , piece_id=HS23LZ37LL6QR5D2Q4WFE7W5VQPFCOV2MQNM3PTUCMPCJ2VNDHLQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.366070103 (915h14m15.237153148s) [noise=true , address=45.91.116.8:4048 , piece_id=65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.367535814 (915h14m15.238618859s) [noise=true , address=145.79.227.48:4301 , piece_id=HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.369525412 (915h14m15.240608457s) [noise=true , address=45.91.116.6:4974 , piece_id=FRJDOIXXLZMS3FW4X4XAYAIOHTDZ5B7TWCGGM2DY5LTV5RZGU6HQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.371063701 (915h14m15.242146736s) [noise=true , address=145.79.227.2:4115 , piece_id=PB2JJEZEXSLTXW5QTHA6IWT32UMAI5EM3S5TL2CGP7A6A3NLA3GA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.372642217 (915h14m15.243725262s) [noise=true , address=77.90.31.11:4554 , piece_id=H7TROB4ZL5AENH4XYEE3SIFZFXDOE6PKO3AQUA2MACPQYLWJ4XPQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:17.184597103 (915h14m17.055680148s) [upload=801054]: Next returned nil segment Jan 12 18:43:17.184643092 (915h14m17.055726127s) [upload=801054]: Waiting for error group managing segments... Jan 12 18:43:17.495324372 (915h14m17.366407407s) [noise=true , address=145.79.227.49:4337 , piece_id=EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.525651226 (915h14m17.396734261s) [noise=true , address=145.79.227.48:4301 , piece_id=HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.536956361 (915h14m17.408039396s) [noise=true , address=145.79.227.16:4909 , piece_id=HS23LZ37LL6QR5D2Q4WFE7W5VQPFCOV2MQNM3PTUCMPCJ2VNDHLQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.624655888 (915h14m17.495738933s) [noise=true , address=45.91.116.6:4974 , piece_id=FRJDOIXXLZMS3FW4X4XAYAIOHTDZ5B7TWCGGM2DY5LTV5RZGU6HQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.629752162 (915h14m17.500835197s) [noise=true , address=45.91.116.7:4012 , piece_id=YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.645478456 (915h14m17.516561491s) [noise=true , address=145.79.227.2:4115 , piece_id=PB2JJEZEXSLTXW5QTHA6IWT32UMAI5EM3S5TL2CGP7A6A3NLA3GA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.676056038 (915h14m17.547139083s) [noise=true , address=45.91.116.14:4189 , piece_id=I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.698909442 (915h14m17.569992488s) [noise=true , address=45.91.116.8:4048 , piece_id=65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.721440313 (915h14m17.592523348s) [noise=true , address=77.90.31.11:4554 , piece_id=H7TROB4ZL5AENH4XYEE3SIFZFXDOE6PKO3AQUA2MACPQYLWJ4XPQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.918612414 (915h14m17.789695459s) [noise=true , address=145.79.227.44:4194 , piece_id=PKZFIIDQYD5O2P25BKIMA45XLXVXZQSLV5H7A2JJGG7XOAHOO2QA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.518838258 (915h14m18.389921293s) [noise=true , address=145.79.227.29:4943 , piece_id=EHLLIAZBTJOBDU2VLA2JXDVQK67WSCRU3FR24HBWK2QF2A5XUERA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.721493959 (915h14m18.592576994s) [noise=true , address=77.90.31.19:4841 , piece_id=PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.727505725 (915h14m18.59858876s) [noise=true , address=77.90.31.12:4590 , piece_id=LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.727530473 (915h14m18.598613518s) [seg_pos={0 0}, upload=801054]: Segment reached optimal threshold of 13 pieces. Jan 12 18:43:18.727812850 (915h14m18.598895885s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.727825525 (915h14m18.59890856s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728214255 (915h14m18.59929729s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.728221819 (915h14m18.599304854s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728386072 (915h14m18.599469107s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.728392494 (915h14m18.599475529s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728413945 (915h14m18.59949698s) [seg_pos={0 0}, upload=801054]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 18:43:18.728512563 (915h14m18.599595598s) [upload=801054]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 18:43:18.728518674 (915h14m18.599601719s) [upload=801054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 18:43:19.041837336 (915h14m18.912920371s) [upload=801054]: Done uploading segments... Jan 12 18:43:19.041856071 (915h14m18.912939116s) [upload=801054]: Upload finished. err: <nil> Duration: 5.825320655s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [.X.XXXXXX.X.XXX.X.XXXXX.XX....X.XXX.X..X.X.X....XX..........] duration: 4.089178247s trace: https://jaeger.dev.storj.io/trace/64e8accbe0a8a091 FAIL (2026-01-12 18:51:13 UTC) orbiter warning: context deadline exceeded Duration: 4.089370914s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.761467858s trace: https://jaeger.dev.storj.io/trace/de69fa036d8b02 FAIL (2026-01-12 18:50:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ) context deadline exceeded; piecestore: (Node ID: 1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE, Piece ID: EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ) context deadline exceeded; piecestore: (Node ID: 18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29, Piece ID: HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ) context deadline exceeded ======LOGS====== Jan 12 18:43:13.858719809 (915h14m13.729802844s) [upload=801054]: Starting upload Jan 12 18:43:13.858805642 (915h14m13.729888677s) [upload=801054]: Done starting upload Jan 12 18:43:13.858880885 (915h14m13.72996393s) [upload=801054]: Uploading segments... Jan 12 18:43:13.859066038 (915h14m13.730149083s) [upload=801054]: Got next segment. Inline: false Jan 12 18:43:13.859125010 (915h14m13.730208045s) [upload=801054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 18:43:13.859131923 (915h14m13.730214968s) [upload=801054]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 18:43:15.358668982 (915h14m15.229752017s) [seg_pos={0 0}, upload=801054]: Begin upload segment... Jan 12 18:43:15.358714368 (915h14m15.229797413s) [seg_pos={0 0}, upload=801054]: Done begin upload segment. Jan 12 18:43:15.358745317 (915h14m15.229828352s) [noise=true , address=77.90.31.12:4590 , piece_id=LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.358908408 (915h14m15.229991453s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.359387038 (915h14m15.230470083s) [noise=true , address=145.79.227.44:4194 , piece_id=PKZFIIDQYD5O2P25BKIMA45XLXVXZQSLV5H7A2JJGG7XOAHOO2QA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.359851062 (915h14m15.230934107s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360286561 (915h14m15.231369606s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360600609 (915h14m15.231683654s) [noise=true , address=145.79.227.29:4943 , piece_id=EHLLIAZBTJOBDU2VLA2JXDVQK67WSCRU3FR24HBWK2QF2A5XUERA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360877186 (915h14m15.231960231s) [noise=true , address=77.90.31.19:4841 , piece_id=PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.361298538 (915h14m15.232381573s) [noise=true , address=145.79.227.49:4337 , piece_id=EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.362993415 (915h14m15.23407645s) [noise=true , address=45.91.116.14:4189 , piece_id=I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.363095780 (915h14m15.234178825s) [noise=true , address=45.91.116.7:4012 , piece_id=YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.364580467 (915h14m15.235663502s) [noise=true , address=145.79.227.16:4909 , piece_id=HS23LZ37LL6QR5D2Q4WFE7W5VQPFCOV2MQNM3PTUCMPCJ2VNDHLQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.366070103 (915h14m15.237153148s) [noise=true , address=45.91.116.8:4048 , piece_id=65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.367535814 (915h14m15.238618859s) [noise=true , address=145.79.227.48:4301 , piece_id=HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.369525412 (915h14m15.240608457s) [noise=true , address=45.91.116.6:4974 , piece_id=FRJDOIXXLZMS3FW4X4XAYAIOHTDZ5B7TWCGGM2DY5LTV5RZGU6HQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.371063701 (915h14m15.242146736s) [noise=true , address=145.79.227.2:4115 , piece_id=PB2JJEZEXSLTXW5QTHA6IWT32UMAI5EM3S5TL2CGP7A6A3NLA3GA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.372642217 (915h14m15.243725262s) [noise=true , address=77.90.31.11:4554 , piece_id=H7TROB4ZL5AENH4XYEE3SIFZFXDOE6PKO3AQUA2MACPQYLWJ4XPQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:17.184597103 (915h14m17.055680148s) [upload=801054]: Next returned nil segment Jan 12 18:43:17.184643092 (915h14m17.055726127s) [upload=801054]: Waiting for error group managing segments... Jan 12 18:43:17.495324372 (915h14m17.366407407s) [noise=true , address=145.79.227.49:4337 , piece_id=EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.525651226 (915h14m17.396734261s) [noise=true , address=145.79.227.48:4301 , piece_id=HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.536956361 (915h14m17.408039396s) [noise=true , address=145.79.227.16:4909 , piece_id=HS23LZ37LL6QR5D2Q4WFE7W5VQPFCOV2MQNM3PTUCMPCJ2VNDHLQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.624655888 (915h14m17.495738933s) [noise=true , address=45.91.116.6:4974 , piece_id=FRJDOIXXLZMS3FW4X4XAYAIOHTDZ5B7TWCGGM2DY5LTV5RZGU6HQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.629752162 (915h14m17.500835197s) [noise=true , address=45.91.116.7:4012 , piece_id=YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.645478456 (915h14m17.516561491s) [noise=true , address=145.79.227.2:4115 , piece_id=PB2JJEZEXSLTXW5QTHA6IWT32UMAI5EM3S5TL2CGP7A6A3NLA3GA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.676056038 (915h14m17.547139083s) [noise=true , address=45.91.116.14:4189 , piece_id=I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.698909442 (915h14m17.569992488s) [noise=true , address=45.91.116.8:4048 , piece_id=65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.721440313 (915h14m17.592523348s) [noise=true , address=77.90.31.11:4554 , piece_id=H7TROB4ZL5AENH4XYEE3SIFZFXDOE6PKO3AQUA2MACPQYLWJ4XPQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.918612414 (915h14m17.789695459s) [noise=true , address=145.79.227.44:4194 , piece_id=PKZFIIDQYD5O2P25BKIMA45XLXVXZQSLV5H7A2JJGG7XOAHOO2QA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.518838258 (915h14m18.389921293s) [noise=true , address=145.79.227.29:4943 , piece_id=EHLLIAZBTJOBDU2VLA2JXDVQK67WSCRU3FR24HBWK2QF2A5XUERA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.721493959 (915h14m18.592576994s) [noise=true , address=77.90.31.19:4841 , piece_id=PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.727505725 (915h14m18.59858876s) [noise=true , address=77.90.31.12:4590 , piece_id=LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.727530473 (915h14m18.598613518s) [seg_pos={0 0}, upload=801054]: Segment reached optimal threshold of 13 pieces. Jan 12 18:43:18.727812850 (915h14m18.598895885s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.727825525 (915h14m18.59890856s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728214255 (915h14m18.59929729s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.728221819 (915h14m18.599304854s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728386072 (915h14m18.599469107s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.728392494 (915h14m18.599475529s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728413945 (915h14m18.59949698s) [seg_pos={0 0}, upload=801054]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 18:43:18.728512563 (915h14m18.599595598s) [upload=801054]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 18:43:18.728518674 (915h14m18.599601719s) [upload=801054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 18:43:19.041837336 (915h14m18.912920371s) [upload=801054]: Done uploading segments... Jan 12 18:43:19.041856071 (915h14m18.912939116s) [upload=801054]: Upload finished. err: <nil> Duration: 5.761499107s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.933939109s trace: https://jaeger.dev.storj.io/trace/de69fa036d8b02 FAIL (2026-01-12 18:50:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ) context deadline exceeded; piecestore: (Node ID: 1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE, Piece ID: EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ) context deadline exceeded; piecestore: (Node ID: 18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29, Piece ID: HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ) context deadline exceeded ======LOGS====== Jan 12 18:43:13.858719809 (915h14m13.729802844s) [upload=801054]: Starting upload Jan 12 18:43:13.858805642 (915h14m13.729888677s) [upload=801054]: Done starting upload Jan 12 18:43:13.858880885 (915h14m13.72996393s) [upload=801054]: Uploading segments... Jan 12 18:43:13.859066038 (915h14m13.730149083s) [upload=801054]: Got next segment. Inline: false Jan 12 18:43:13.859125010 (915h14m13.730208045s) [upload=801054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 18:43:13.859131923 (915h14m13.730214968s) [upload=801054]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 18:43:15.358668982 (915h14m15.229752017s) [seg_pos={0 0}, upload=801054]: Begin upload segment... Jan 12 18:43:15.358714368 (915h14m15.229797413s) [seg_pos={0 0}, upload=801054]: Done begin upload segment. Jan 12 18:43:15.358745317 (915h14m15.229828352s) [noise=true , address=77.90.31.12:4590 , piece_id=LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.358908408 (915h14m15.229991453s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.359387038 (915h14m15.230470083s) [noise=true , address=145.79.227.44:4194 , piece_id=PKZFIIDQYD5O2P25BKIMA45XLXVXZQSLV5H7A2JJGG7XOAHOO2QA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.359851062 (915h14m15.230934107s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360286561 (915h14m15.231369606s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360600609 (915h14m15.231683654s) [noise=true , address=145.79.227.29:4943 , piece_id=EHLLIAZBTJOBDU2VLA2JXDVQK67WSCRU3FR24HBWK2QF2A5XUERA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360877186 (915h14m15.231960231s) [noise=true , address=77.90.31.19:4841 , piece_id=PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.361298538 (915h14m15.232381573s) [noise=true , address=145.79.227.49:4337 , piece_id=EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.362993415 (915h14m15.23407645s) [noise=true , address=45.91.116.14:4189 , piece_id=I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.363095780 (915h14m15.234178825s) [noise=true , address=45.91.116.7:4012 , piece_id=YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.364580467 (915h14m15.235663502s) [noise=true , address=145.79.227.16:4909 , piece_id=HS23LZ37LL6QR5D2Q4WFE7W5VQPFCOV2MQNM3PTUCMPCJ2VNDHLQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.366070103 (915h14m15.237153148s) [noise=true , address=45.91.116.8:4048 , piece_id=65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.367535814 (915h14m15.238618859s) [noise=true , address=145.79.227.48:4301 , piece_id=HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.369525412 (915h14m15.240608457s) [noise=true , address=45.91.116.6:4974 , piece_id=FRJDOIXXLZMS3FW4X4XAYAIOHTDZ5B7TWCGGM2DY5LTV5RZGU6HQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.371063701 (915h14m15.242146736s) [noise=true , address=145.79.227.2:4115 , piece_id=PB2JJEZEXSLTXW5QTHA6IWT32UMAI5EM3S5TL2CGP7A6A3NLA3GA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.372642217 (915h14m15.243725262s) [noise=true , address=77.90.31.11:4554 , piece_id=H7TROB4ZL5AENH4XYEE3SIFZFXDOE6PKO3AQUA2MACPQYLWJ4XPQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:17.184597103 (915h14m17.055680148s) [upload=801054]: Next returned nil segment Jan 12 18:43:17.184643092 (915h14m17.055726127s) [upload=801054]: Waiting for error group managing segments... Jan 12 18:43:17.495324372 (915h14m17.366407407s) [noise=true , address=145.79.227.49:4337 , piece_id=EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.525651226 (915h14m17.396734261s) [noise=true , address=145.79.227.48:4301 , piece_id=HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.536956361 (915h14m17.408039396s) [noise=true , address=145.79.227.16:4909 , piece_id=HS23LZ37LL6QR5D2Q4WFE7W5VQPFCOV2MQNM3PTUCMPCJ2VNDHLQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.624655888 (915h14m17.495738933s) [noise=true , address=45.91.116.6:4974 , piece_id=FRJDOIXXLZMS3FW4X4XAYAIOHTDZ5B7TWCGGM2DY5LTV5RZGU6HQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.629752162 (915h14m17.500835197s) [noise=true , address=45.91.116.7:4012 , piece_id=YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.645478456 (915h14m17.516561491s) [noise=true , address=145.79.227.2:4115 , piece_id=PB2JJEZEXSLTXW5QTHA6IWT32UMAI5EM3S5TL2CGP7A6A3NLA3GA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.676056038 (915h14m17.547139083s) [noise=true , address=45.91.116.14:4189 , piece_id=I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.698909442 (915h14m17.569992488s) [noise=true , address=45.91.116.8:4048 , piece_id=65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.721440313 (915h14m17.592523348s) [noise=true , address=77.90.31.11:4554 , piece_id=H7TROB4ZL5AENH4XYEE3SIFZFXDOE6PKO3AQUA2MACPQYLWJ4XPQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.918612414 (915h14m17.789695459s) [noise=true , address=145.79.227.44:4194 , piece_id=PKZFIIDQYD5O2P25BKIMA45XLXVXZQSLV5H7A2JJGG7XOAHOO2QA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.518838258 (915h14m18.389921293s) [noise=true , address=145.79.227.29:4943 , piece_id=EHLLIAZBTJOBDU2VLA2JXDVQK67WSCRU3FR24HBWK2QF2A5XUERA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.721493959 (915h14m18.592576994s) [noise=true , address=77.90.31.19:4841 , piece_id=PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.727505725 (915h14m18.59858876s) [noise=true , address=77.90.31.12:4590 , piece_id=LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.727530473 (915h14m18.598613518s) [seg_pos={0 0}, upload=801054]: Segment reached optimal threshold of 13 pieces. Jan 12 18:43:18.727812850 (915h14m18.598895885s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.727825525 (915h14m18.59890856s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728214255 (915h14m18.59929729s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.728221819 (915h14m18.599304854s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728386072 (915h14m18.599469107s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.728392494 (915h14m18.599475529s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728413945 (915h14m18.59949698s) [seg_pos={0 0}, upload=801054]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 18:43:18.728512563 (915h14m18.599595598s) [upload=801054]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 18:43:18.728518674 (915h14m18.599601719s) [upload=801054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 18:43:19.041837336 (915h14m18.912920371s) [upload=801054]: Done uploading segments... Jan 12 18:43:19.041856071 (915h14m18.912939116s) [upload=801054]: Upload finished. err: <nil> Duration: 5.933971379s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 29 out of 60 is failed (threshold=6) [..X.XXXXXX.X.XXX.X.XXXXX.XX....X.XXX.X..X.X.X....XX.........] duration: 5.002231799s trace: https://jaeger.dev.storj.io/trace/64e8accbe0a8a091 FAIL (2026-01-12 18:50:33 UTC) orbiter warning: context deadline exceeded Duration: 5.002258239s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.863376153s trace: https://jaeger.dev.storj.io/trace/de69fa036d8b02 FAIL (2026-01-12 18:49:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ) context deadline exceeded; piecestore: (Node ID: 1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE, Piece ID: EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ) context deadline exceeded; piecestore: (Node ID: 18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29, Piece ID: HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ) context deadline exceeded ======LOGS====== Jan 12 18:43:13.858719809 (915h14m13.729802844s) [upload=801054]: Starting upload Jan 12 18:43:13.858805642 (915h14m13.729888677s) [upload=801054]: Done starting upload Jan 12 18:43:13.858880885 (915h14m13.72996393s) [upload=801054]: Uploading segments... Jan 12 18:43:13.859066038 (915h14m13.730149083s) [upload=801054]: Got next segment. Inline: false Jan 12 18:43:13.859125010 (915h14m13.730208045s) [upload=801054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 18:43:13.859131923 (915h14m13.730214968s) [upload=801054]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 18:43:15.358668982 (915h14m15.229752017s) [seg_pos={0 0}, upload=801054]: Begin upload segment... Jan 12 18:43:15.358714368 (915h14m15.229797413s) [seg_pos={0 0}, upload=801054]: Done begin upload segment. Jan 12 18:43:15.358745317 (915h14m15.229828352s) [noise=true , address=77.90.31.12:4590 , piece_id=LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.358908408 (915h14m15.229991453s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.359387038 (915h14m15.230470083s) [noise=true , address=145.79.227.44:4194 , piece_id=PKZFIIDQYD5O2P25BKIMA45XLXVXZQSLV5H7A2JJGG7XOAHOO2QA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.359851062 (915h14m15.230934107s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360286561 (915h14m15.231369606s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360600609 (915h14m15.231683654s) [noise=true , address=145.79.227.29:4943 , piece_id=EHLLIAZBTJOBDU2VLA2JXDVQK67WSCRU3FR24HBWK2QF2A5XUERA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360877186 (915h14m15.231960231s) [noise=true , address=77.90.31.19:4841 , piece_id=PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.361298538 (915h14m15.232381573s) [noise=true , address=145.79.227.49:4337 , piece_id=EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.362993415 (915h14m15.23407645s) [noise=true , address=45.91.116.14:4189 , piece_id=I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.363095780 (915h14m15.234178825s) [noise=true , address=45.91.116.7:4012 , piece_id=YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.364580467 (915h14m15.235663502s) [noise=true , address=145.79.227.16:4909 , piece_id=HS23LZ37LL6QR5D2Q4WFE7W5VQPFCOV2MQNM3PTUCMPCJ2VNDHLQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.366070103 (915h14m15.237153148s) [noise=true , address=45.91.116.8:4048 , piece_id=65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.367535814 (915h14m15.238618859s) [noise=true , address=145.79.227.48:4301 , piece_id=HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.369525412 (915h14m15.240608457s) [noise=true , address=45.91.116.6:4974 , piece_id=FRJDOIXXLZMS3FW4X4XAYAIOHTDZ5B7TWCGGM2DY5LTV5RZGU6HQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.371063701 (915h14m15.242146736s) [noise=true , address=145.79.227.2:4115 , piece_id=PB2JJEZEXSLTXW5QTHA6IWT32UMAI5EM3S5TL2CGP7A6A3NLA3GA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.372642217 (915h14m15.243725262s) [noise=true , address=77.90.31.11:4554 , piece_id=H7TROB4ZL5AENH4XYEE3SIFZFXDOE6PKO3AQUA2MACPQYLWJ4XPQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:17.184597103 (915h14m17.055680148s) [upload=801054]: Next returned nil segment Jan 12 18:43:17.184643092 (915h14m17.055726127s) [upload=801054]: Waiting for error group managing segments... Jan 12 18:43:17.495324372 (915h14m17.366407407s) [noise=true , address=145.79.227.49:4337 , piece_id=EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.525651226 (915h14m17.396734261s) [noise=true , address=145.79.227.48:4301 , piece_id=HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.536956361 (915h14m17.408039396s) [noise=true , address=145.79.227.16:4909 , piece_id=HS23LZ37LL6QR5D2Q4WFE7W5VQPFCOV2MQNM3PTUCMPCJ2VNDHLQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.624655888 (915h14m17.495738933s) [noise=true , address=45.91.116.6:4974 , piece_id=FRJDOIXXLZMS3FW4X4XAYAIOHTDZ5B7TWCGGM2DY5LTV5RZGU6HQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.629752162 (915h14m17.500835197s) [noise=true , address=45.91.116.7:4012 , piece_id=YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.645478456 (915h14m17.516561491s) [noise=true , address=145.79.227.2:4115 , piece_id=PB2JJEZEXSLTXW5QTHA6IWT32UMAI5EM3S5TL2CGP7A6A3NLA3GA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.676056038 (915h14m17.547139083s) [noise=true , address=45.91.116.14:4189 , piece_id=I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.698909442 (915h14m17.569992488s) [noise=true , address=45.91.116.8:4048 , piece_id=65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.721440313 (915h14m17.592523348s) [noise=true , address=77.90.31.11:4554 , piece_id=H7TROB4ZL5AENH4XYEE3SIFZFXDOE6PKO3AQUA2MACPQYLWJ4XPQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.918612414 (915h14m17.789695459s) [noise=true , address=145.79.227.44:4194 , piece_id=PKZFIIDQYD5O2P25BKIMA45XLXVXZQSLV5H7A2JJGG7XOAHOO2QA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.518838258 (915h14m18.389921293s) [noise=true , address=145.79.227.29:4943 , piece_id=EHLLIAZBTJOBDU2VLA2JXDVQK67WSCRU3FR24HBWK2QF2A5XUERA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.721493959 (915h14m18.592576994s) [noise=true , address=77.90.31.19:4841 , piece_id=PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.727505725 (915h14m18.59858876s) [noise=true , address=77.90.31.12:4590 , piece_id=LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.727530473 (915h14m18.598613518s) [seg_pos={0 0}, upload=801054]: Segment reached optimal threshold of 13 pieces. Jan 12 18:43:18.727812850 (915h14m18.598895885s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.727825525 (915h14m18.59890856s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728214255 (915h14m18.59929729s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.728221819 (915h14m18.599304854s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728386072 (915h14m18.599469107s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.728392494 (915h14m18.599475529s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728413945 (915h14m18.59949698s) [seg_pos={0 0}, upload=801054]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 18:43:18.728512563 (915h14m18.599595598s) [upload=801054]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 18:43:18.728518674 (915h14m18.599601719s) [upload=801054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 18:43:19.041837336 (915h14m18.912920371s) [upload=801054]: Done uploading segments... Jan 12 18:43:19.041856071 (915h14m18.912939116s) [upload=801054]: Upload finished. err: <nil> Duration: 5.863397452s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 30 out of 60 is failed (threshold=6) [X..X.XXXXXX.X.XXX.X.XXXXX.XX....X.XXX.X..X.X.X....XX........] duration: 4.283539314s trace: https://jaeger.dev.storj.io/trace/64e8accbe0a8a091 FAIL (2026-01-12 18:49:44 UTC) orbiter warning: context deadline exceeded Duration: 4.283569119s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.861834475s trace: https://jaeger.dev.storj.io/trace/de69fa036d8b02 FAIL (2026-01-12 18:49:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ) context deadline exceeded; piecestore: (Node ID: 1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE, Piece ID: EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ) context deadline exceeded; piecestore: (Node ID: 18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29, Piece ID: HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ) context deadline exceeded ======LOGS====== Jan 12 18:43:13.858719809 (915h14m13.729802844s) [upload=801054]: Starting upload Jan 12 18:43:13.858805642 (915h14m13.729888677s) [upload=801054]: Done starting upload Jan 12 18:43:13.858880885 (915h14m13.72996393s) [upload=801054]: Uploading segments... Jan 12 18:43:13.859066038 (915h14m13.730149083s) [upload=801054]: Got next segment. Inline: false Jan 12 18:43:13.859125010 (915h14m13.730208045s) [upload=801054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 18:43:13.859131923 (915h14m13.730214968s) [upload=801054]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 18:43:15.358668982 (915h14m15.229752017s) [seg_pos={0 0}, upload=801054]: Begin upload segment... Jan 12 18:43:15.358714368 (915h14m15.229797413s) [seg_pos={0 0}, upload=801054]: Done begin upload segment. Jan 12 18:43:15.358745317 (915h14m15.229828352s) [noise=true , address=77.90.31.12:4590 , piece_id=LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.358908408 (915h14m15.229991453s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.359387038 (915h14m15.230470083s) [noise=true , address=145.79.227.44:4194 , piece_id=PKZFIIDQYD5O2P25BKIMA45XLXVXZQSLV5H7A2JJGG7XOAHOO2QA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.359851062 (915h14m15.230934107s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360286561 (915h14m15.231369606s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360600609 (915h14m15.231683654s) [noise=true , address=145.79.227.29:4943 , piece_id=EHLLIAZBTJOBDU2VLA2JXDVQK67WSCRU3FR24HBWK2QF2A5XUERA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360877186 (915h14m15.231960231s) [noise=true , address=77.90.31.19:4841 , piece_id=PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.361298538 (915h14m15.232381573s) [noise=true , address=145.79.227.49:4337 , piece_id=EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.362993415 (915h14m15.23407645s) [noise=true , address=45.91.116.14:4189 , piece_id=I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.363095780 (915h14m15.234178825s) [noise=true , address=45.91.116.7:4012 , piece_id=YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.364580467 (915h14m15.235663502s) [noise=true , address=145.79.227.16:4909 , piece_id=HS23LZ37LL6QR5D2Q4WFE7W5VQPFCOV2MQNM3PTUCMPCJ2VNDHLQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.366070103 (915h14m15.237153148s) [noise=true , address=45.91.116.8:4048 , piece_id=65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.367535814 (915h14m15.238618859s) [noise=true , address=145.79.227.48:4301 , piece_id=HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.369525412 (915h14m15.240608457s) [noise=true , address=45.91.116.6:4974 , piece_id=FRJDOIXXLZMS3FW4X4XAYAIOHTDZ5B7TWCGGM2DY5LTV5RZGU6HQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.371063701 (915h14m15.242146736s) [noise=true , address=145.79.227.2:4115 , piece_id=PB2JJEZEXSLTXW5QTHA6IWT32UMAI5EM3S5TL2CGP7A6A3NLA3GA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.372642217 (915h14m15.243725262s) [noise=true , address=77.90.31.11:4554 , piece_id=H7TROB4ZL5AENH4XYEE3SIFZFXDOE6PKO3AQUA2MACPQYLWJ4XPQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:17.184597103 (915h14m17.055680148s) [upload=801054]: Next returned nil segment Jan 12 18:43:17.184643092 (915h14m17.055726127s) [upload=801054]: Waiting for error group managing segments... Jan 12 18:43:17.495324372 (915h14m17.366407407s) [noise=true , address=145.79.227.49:4337 , piece_id=EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.525651226 (915h14m17.396734261s) [noise=true , address=145.79.227.48:4301 , piece_id=HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.536956361 (915h14m17.408039396s) [noise=true , address=145.79.227.16:4909 , piece_id=HS23LZ37LL6QR5D2Q4WFE7W5VQPFCOV2MQNM3PTUCMPCJ2VNDHLQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.624655888 (915h14m17.495738933s) [noise=true , address=45.91.116.6:4974 , piece_id=FRJDOIXXLZMS3FW4X4XAYAIOHTDZ5B7TWCGGM2DY5LTV5RZGU6HQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.629752162 (915h14m17.500835197s) [noise=true , address=45.91.116.7:4012 , piece_id=YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.645478456 (915h14m17.516561491s) [noise=true , address=145.79.227.2:4115 , piece_id=PB2JJEZEXSLTXW5QTHA6IWT32UMAI5EM3S5TL2CGP7A6A3NLA3GA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.676056038 (915h14m17.547139083s) [noise=true , address=45.91.116.14:4189 , piece_id=I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.698909442 (915h14m17.569992488s) [noise=true , address=45.91.116.8:4048 , piece_id=65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.721440313 (915h14m17.592523348s) [noise=true , address=77.90.31.11:4554 , piece_id=H7TROB4ZL5AENH4XYEE3SIFZFXDOE6PKO3AQUA2MACPQYLWJ4XPQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.918612414 (915h14m17.789695459s) [noise=true , address=145.79.227.44:4194 , piece_id=PKZFIIDQYD5O2P25BKIMA45XLXVXZQSLV5H7A2JJGG7XOAHOO2QA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.518838258 (915h14m18.389921293s) [noise=true , address=145.79.227.29:4943 , piece_id=EHLLIAZBTJOBDU2VLA2JXDVQK67WSCRU3FR24HBWK2QF2A5XUERA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.721493959 (915h14m18.592576994s) [noise=true , address=77.90.31.19:4841 , piece_id=PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.727505725 (915h14m18.59858876s) [noise=true , address=77.90.31.12:4590 , piece_id=LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.727530473 (915h14m18.598613518s) [seg_pos={0 0}, upload=801054]: Segment reached optimal threshold of 13 pieces. Jan 12 18:43:18.727812850 (915h14m18.598895885s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.727825525 (915h14m18.59890856s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728214255 (915h14m18.59929729s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.728221819 (915h14m18.599304854s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728386072 (915h14m18.599469107s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.728392494 (915h14m18.599475529s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728413945 (915h14m18.59949698s) [seg_pos={0 0}, upload=801054]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 18:43:18.728512563 (915h14m18.599595598s) [upload=801054]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 18:43:18.728518674 (915h14m18.599601719s) [upload=801054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 18:43:19.041837336 (915h14m18.912920371s) [upload=801054]: Done uploading segments... Jan 12 18:43:19.041856071 (915h14m18.912939116s) [upload=801054]: Upload finished. err: <nil> Duration: 5.861861045s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [XX..X.XXXXXX.X.XXX.X.XXXXX.XX....X.XXX.X..X.X.X....XX.......] duration: 3.778770428s trace: https://jaeger.dev.storj.io/trace/64e8accbe0a8a091 FAIL (2026-01-12 18:48:39 UTC) orbiter warning: context deadline exceeded Duration: 3.77879237s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 6.474697734s trace: https://jaeger.dev.storj.io/trace/de69fa036d8b02 FAIL (2026-01-12 18:48:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ) context deadline exceeded; piecestore: (Node ID: 1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE, Piece ID: EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ) context deadline exceeded; piecestore: (Node ID: 18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29, Piece ID: HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ) context deadline exceeded ======LOGS====== Jan 12 18:43:13.858719809 (915h14m13.729802844s) [upload=801054]: Starting upload Jan 12 18:43:13.858805642 (915h14m13.729888677s) [upload=801054]: Done starting upload Jan 12 18:43:13.858880885 (915h14m13.72996393s) [upload=801054]: Uploading segments... Jan 12 18:43:13.859066038 (915h14m13.730149083s) [upload=801054]: Got next segment. Inline: false Jan 12 18:43:13.859125010 (915h14m13.730208045s) [upload=801054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 18:43:13.859131923 (915h14m13.730214968s) [upload=801054]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 18:43:15.358668982 (915h14m15.229752017s) [seg_pos={0 0}, upload=801054]: Begin upload segment... Jan 12 18:43:15.358714368 (915h14m15.229797413s) [seg_pos={0 0}, upload=801054]: Done begin upload segment. Jan 12 18:43:15.358745317 (915h14m15.229828352s) [noise=true , address=77.90.31.12:4590 , piece_id=LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.358908408 (915h14m15.229991453s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.359387038 (915h14m15.230470083s) [noise=true , address=145.79.227.44:4194 , piece_id=PKZFIIDQYD5O2P25BKIMA45XLXVXZQSLV5H7A2JJGG7XOAHOO2QA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.359851062 (915h14m15.230934107s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360286561 (915h14m15.231369606s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360600609 (915h14m15.231683654s) [noise=true , address=145.79.227.29:4943 , piece_id=EHLLIAZBTJOBDU2VLA2JXDVQK67WSCRU3FR24HBWK2QF2A5XUERA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360877186 (915h14m15.231960231s) [noise=true , address=77.90.31.19:4841 , piece_id=PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.361298538 (915h14m15.232381573s) [noise=true , address=145.79.227.49:4337 , piece_id=EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.362993415 (915h14m15.23407645s) [noise=true , address=45.91.116.14:4189 , piece_id=I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.363095780 (915h14m15.234178825s) [noise=true , address=45.91.116.7:4012 , piece_id=YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.364580467 (915h14m15.235663502s) [noise=true , address=145.79.227.16:4909 , piece_id=HS23LZ37LL6QR5D2Q4WFE7W5VQPFCOV2MQNM3PTUCMPCJ2VNDHLQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.366070103 (915h14m15.237153148s) [noise=true , address=45.91.116.8:4048 , piece_id=65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.367535814 (915h14m15.238618859s) [noise=true , address=145.79.227.48:4301 , piece_id=HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.369525412 (915h14m15.240608457s) [noise=true , address=45.91.116.6:4974 , piece_id=FRJDOIXXLZMS3FW4X4XAYAIOHTDZ5B7TWCGGM2DY5LTV5RZGU6HQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.371063701 (915h14m15.242146736s) [noise=true , address=145.79.227.2:4115 , piece_id=PB2JJEZEXSLTXW5QTHA6IWT32UMAI5EM3S5TL2CGP7A6A3NLA3GA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.372642217 (915h14m15.243725262s) [noise=true , address=77.90.31.11:4554 , piece_id=H7TROB4ZL5AENH4XYEE3SIFZFXDOE6PKO3AQUA2MACPQYLWJ4XPQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:17.184597103 (915h14m17.055680148s) [upload=801054]: Next returned nil segment Jan 12 18:43:17.184643092 (915h14m17.055726127s) [upload=801054]: Waiting for error group managing segments... Jan 12 18:43:17.495324372 (915h14m17.366407407s) [noise=true , address=145.79.227.49:4337 , piece_id=EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.525651226 (915h14m17.396734261s) [noise=true , address=145.79.227.48:4301 , piece_id=HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.536956361 (915h14m17.408039396s) [noise=true , address=145.79.227.16:4909 , piece_id=HS23LZ37LL6QR5D2Q4WFE7W5VQPFCOV2MQNM3PTUCMPCJ2VNDHLQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.624655888 (915h14m17.495738933s) [noise=true , address=45.91.116.6:4974 , piece_id=FRJDOIXXLZMS3FW4X4XAYAIOHTDZ5B7TWCGGM2DY5LTV5RZGU6HQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.629752162 (915h14m17.500835197s) [noise=true , address=45.91.116.7:4012 , piece_id=YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.645478456 (915h14m17.516561491s) [noise=true , address=145.79.227.2:4115 , piece_id=PB2JJEZEXSLTXW5QTHA6IWT32UMAI5EM3S5TL2CGP7A6A3NLA3GA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.676056038 (915h14m17.547139083s) [noise=true , address=45.91.116.14:4189 , piece_id=I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.698909442 (915h14m17.569992488s) [noise=true , address=45.91.116.8:4048 , piece_id=65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.721440313 (915h14m17.592523348s) [noise=true , address=77.90.31.11:4554 , piece_id=H7TROB4ZL5AENH4XYEE3SIFZFXDOE6PKO3AQUA2MACPQYLWJ4XPQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.918612414 (915h14m17.789695459s) [noise=true , address=145.79.227.44:4194 , piece_id=PKZFIIDQYD5O2P25BKIMA45XLXVXZQSLV5H7A2JJGG7XOAHOO2QA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.518838258 (915h14m18.389921293s) [noise=true , address=145.79.227.29:4943 , piece_id=EHLLIAZBTJOBDU2VLA2JXDVQK67WSCRU3FR24HBWK2QF2A5XUERA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.721493959 (915h14m18.592576994s) [noise=true , address=77.90.31.19:4841 , piece_id=PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.727505725 (915h14m18.59858876s) [noise=true , address=77.90.31.12:4590 , piece_id=LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.727530473 (915h14m18.598613518s) [seg_pos={0 0}, upload=801054]: Segment reached optimal threshold of 13 pieces. Jan 12 18:43:18.727812850 (915h14m18.598895885s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.727825525 (915h14m18.59890856s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728214255 (915h14m18.59929729s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.728221819 (915h14m18.599304854s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728386072 (915h14m18.599469107s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.728392494 (915h14m18.599475529s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728413945 (915h14m18.59949698s) [seg_pos={0 0}, upload=801054]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 18:43:18.728512563 (915h14m18.599595598s) [upload=801054]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 18:43:18.728518674 (915h14m18.599601719s) [upload=801054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 18:43:19.041837336 (915h14m18.912920371s) [upload=801054]: Done uploading segments... Jan 12 18:43:19.041856071 (915h14m18.912939116s) [upload=801054]: Upload finished. err: <nil> Duration: 6.47472163s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 31 out of 60 is failed (threshold=6) [.XX..X.XXXXXX.X.XXX.X.XXXXX.XX....X.XXX.X..X.X.X....XX......] duration: 5.000294692s trace: https://jaeger.dev.storj.io/trace/64e8accbe0a8a091 FAIL (2026-01-12 18:47:33 UTC) orbiter warning: context deadline exceeded Duration: 5.000317006s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.877799628s trace: https://jaeger.dev.storj.io/trace/de69fa036d8b02 FAIL (2026-01-12 18:47:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ) context deadline exceeded; piecestore: (Node ID: 1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE, Piece ID: EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ) context deadline exceeded; piecestore: (Node ID: 18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29, Piece ID: HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ) context deadline exceeded ======LOGS====== Jan 12 18:43:13.858719809 (915h14m13.729802844s) [upload=801054]: Starting upload Jan 12 18:43:13.858805642 (915h14m13.729888677s) [upload=801054]: Done starting upload Jan 12 18:43:13.858880885 (915h14m13.72996393s) [upload=801054]: Uploading segments... Jan 12 18:43:13.859066038 (915h14m13.730149083s) [upload=801054]: Got next segment. Inline: false Jan 12 18:43:13.859125010 (915h14m13.730208045s) [upload=801054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 18:43:13.859131923 (915h14m13.730214968s) [upload=801054]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 18:43:15.358668982 (915h14m15.229752017s) [seg_pos={0 0}, upload=801054]: Begin upload segment... Jan 12 18:43:15.358714368 (915h14m15.229797413s) [seg_pos={0 0}, upload=801054]: Done begin upload segment. Jan 12 18:43:15.358745317 (915h14m15.229828352s) [noise=true , address=77.90.31.12:4590 , piece_id=LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.358908408 (915h14m15.229991453s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.359387038 (915h14m15.230470083s) [noise=true , address=145.79.227.44:4194 , piece_id=PKZFIIDQYD5O2P25BKIMA45XLXVXZQSLV5H7A2JJGG7XOAHOO2QA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.359851062 (915h14m15.230934107s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360286561 (915h14m15.231369606s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360600609 (915h14m15.231683654s) [noise=true , address=145.79.227.29:4943 , piece_id=EHLLIAZBTJOBDU2VLA2JXDVQK67WSCRU3FR24HBWK2QF2A5XUERA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360877186 (915h14m15.231960231s) [noise=true , address=77.90.31.19:4841 , piece_id=PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.361298538 (915h14m15.232381573s) [noise=true , address=145.79.227.49:4337 , piece_id=EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.362993415 (915h14m15.23407645s) [noise=true , address=45.91.116.14:4189 , piece_id=I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.363095780 (915h14m15.234178825s) [noise=true , address=45.91.116.7:4012 , piece_id=YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.364580467 (915h14m15.235663502s) [noise=true , address=145.79.227.16:4909 , piece_id=HS23LZ37LL6QR5D2Q4WFE7W5VQPFCOV2MQNM3PTUCMPCJ2VNDHLQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.366070103 (915h14m15.237153148s) [noise=true , address=45.91.116.8:4048 , piece_id=65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.367535814 (915h14m15.238618859s) [noise=true , address=145.79.227.48:4301 , piece_id=HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.369525412 (915h14m15.240608457s) [noise=true , address=45.91.116.6:4974 , piece_id=FRJDOIXXLZMS3FW4X4XAYAIOHTDZ5B7TWCGGM2DY5LTV5RZGU6HQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.371063701 (915h14m15.242146736s) [noise=true , address=145.79.227.2:4115 , piece_id=PB2JJEZEXSLTXW5QTHA6IWT32UMAI5EM3S5TL2CGP7A6A3NLA3GA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.372642217 (915h14m15.243725262s) [noise=true , address=77.90.31.11:4554 , piece_id=H7TROB4ZL5AENH4XYEE3SIFZFXDOE6PKO3AQUA2MACPQYLWJ4XPQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:17.184597103 (915h14m17.055680148s) [upload=801054]: Next returned nil segment Jan 12 18:43:17.184643092 (915h14m17.055726127s) [upload=801054]: Waiting for error group managing segments... Jan 12 18:43:17.495324372 (915h14m17.366407407s) [noise=true , address=145.79.227.49:4337 , piece_id=EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.525651226 (915h14m17.396734261s) [noise=true , address=145.79.227.48:4301 , piece_id=HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.536956361 (915h14m17.408039396s) [noise=true , address=145.79.227.16:4909 , piece_id=HS23LZ37LL6QR5D2Q4WFE7W5VQPFCOV2MQNM3PTUCMPCJ2VNDHLQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.624655888 (915h14m17.495738933s) [noise=true , address=45.91.116.6:4974 , piece_id=FRJDOIXXLZMS3FW4X4XAYAIOHTDZ5B7TWCGGM2DY5LTV5RZGU6HQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.629752162 (915h14m17.500835197s) [noise=true , address=45.91.116.7:4012 , piece_id=YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.645478456 (915h14m17.516561491s) [noise=true , address=145.79.227.2:4115 , piece_id=PB2JJEZEXSLTXW5QTHA6IWT32UMAI5EM3S5TL2CGP7A6A3NLA3GA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.676056038 (915h14m17.547139083s) [noise=true , address=45.91.116.14:4189 , piece_id=I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.698909442 (915h14m17.569992488s) [noise=true , address=45.91.116.8:4048 , piece_id=65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.721440313 (915h14m17.592523348s) [noise=true , address=77.90.31.11:4554 , piece_id=H7TROB4ZL5AENH4XYEE3SIFZFXDOE6PKO3AQUA2MACPQYLWJ4XPQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.918612414 (915h14m17.789695459s) [noise=true , address=145.79.227.44:4194 , piece_id=PKZFIIDQYD5O2P25BKIMA45XLXVXZQSLV5H7A2JJGG7XOAHOO2QA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.518838258 (915h14m18.389921293s) [noise=true , address=145.79.227.29:4943 , piece_id=EHLLIAZBTJOBDU2VLA2JXDVQK67WSCRU3FR24HBWK2QF2A5XUERA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.721493959 (915h14m18.592576994s) [noise=true , address=77.90.31.19:4841 , piece_id=PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.727505725 (915h14m18.59858876s) [noise=true , address=77.90.31.12:4590 , piece_id=LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.727530473 (915h14m18.598613518s) [seg_pos={0 0}, upload=801054]: Segment reached optimal threshold of 13 pieces. Jan 12 18:43:18.727812850 (915h14m18.598895885s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.727825525 (915h14m18.59890856s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728214255 (915h14m18.59929729s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.728221819 (915h14m18.599304854s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728386072 (915h14m18.599469107s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.728392494 (915h14m18.599475529s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728413945 (915h14m18.59949698s) [seg_pos={0 0}, upload=801054]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 18:43:18.728512563 (915h14m18.599595598s) [upload=801054]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 18:43:18.728518674 (915h14m18.599601719s) [upload=801054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 18:43:19.041837336 (915h14m18.912920371s) [upload=801054]: Done uploading segments... Jan 12 18:43:19.041856071 (915h14m18.912939116s) [upload=801054]: Upload finished. err: <nil> Duration: 5.877823955s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 7.238480745s trace: https://jaeger.dev.storj.io/trace/de69fa036d8b02 FAIL (2026-01-12 18:46:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ) context deadline exceeded; piecestore: (Node ID: 1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE, Piece ID: EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ) context deadline exceeded; piecestore: (Node ID: 18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29, Piece ID: HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ) context deadline exceeded ======LOGS====== Jan 12 18:43:13.858719809 (915h14m13.729802844s) [upload=801054]: Starting upload Jan 12 18:43:13.858805642 (915h14m13.729888677s) [upload=801054]: Done starting upload Jan 12 18:43:13.858880885 (915h14m13.72996393s) [upload=801054]: Uploading segments... Jan 12 18:43:13.859066038 (915h14m13.730149083s) [upload=801054]: Got next segment. Inline: false Jan 12 18:43:13.859125010 (915h14m13.730208045s) [upload=801054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 18:43:13.859131923 (915h14m13.730214968s) [upload=801054]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 18:43:15.358668982 (915h14m15.229752017s) [seg_pos={0 0}, upload=801054]: Begin upload segment... Jan 12 18:43:15.358714368 (915h14m15.229797413s) [seg_pos={0 0}, upload=801054]: Done begin upload segment. Jan 12 18:43:15.358745317 (915h14m15.229828352s) [noise=true , address=77.90.31.12:4590 , piece_id=LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.358908408 (915h14m15.229991453s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.359387038 (915h14m15.230470083s) [noise=true , address=145.79.227.44:4194 , piece_id=PKZFIIDQYD5O2P25BKIMA45XLXVXZQSLV5H7A2JJGG7XOAHOO2QA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.359851062 (915h14m15.230934107s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360286561 (915h14m15.231369606s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360600609 (915h14m15.231683654s) [noise=true , address=145.79.227.29:4943 , piece_id=EHLLIAZBTJOBDU2VLA2JXDVQK67WSCRU3FR24HBWK2QF2A5XUERA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360877186 (915h14m15.231960231s) [noise=true , address=77.90.31.19:4841 , piece_id=PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.361298538 (915h14m15.232381573s) [noise=true , address=145.79.227.49:4337 , piece_id=EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.362993415 (915h14m15.23407645s) [noise=true , address=45.91.116.14:4189 , piece_id=I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.363095780 (915h14m15.234178825s) [noise=true , address=45.91.116.7:4012 , piece_id=YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.364580467 (915h14m15.235663502s) [noise=true , address=145.79.227.16:4909 , piece_id=HS23LZ37LL6QR5D2Q4WFE7W5VQPFCOV2MQNM3PTUCMPCJ2VNDHLQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.366070103 (915h14m15.237153148s) [noise=true , address=45.91.116.8:4048 , piece_id=65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.367535814 (915h14m15.238618859s) [noise=true , address=145.79.227.48:4301 , piece_id=HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.369525412 (915h14m15.240608457s) [noise=true , address=45.91.116.6:4974 , piece_id=FRJDOIXXLZMS3FW4X4XAYAIOHTDZ5B7TWCGGM2DY5LTV5RZGU6HQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.371063701 (915h14m15.242146736s) [noise=true , address=145.79.227.2:4115 , piece_id=PB2JJEZEXSLTXW5QTHA6IWT32UMAI5EM3S5TL2CGP7A6A3NLA3GA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.372642217 (915h14m15.243725262s) [noise=true , address=77.90.31.11:4554 , piece_id=H7TROB4ZL5AENH4XYEE3SIFZFXDOE6PKO3AQUA2MACPQYLWJ4XPQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:17.184597103 (915h14m17.055680148s) [upload=801054]: Next returned nil segment Jan 12 18:43:17.184643092 (915h14m17.055726127s) [upload=801054]: Waiting for error group managing segments... Jan 12 18:43:17.495324372 (915h14m17.366407407s) [noise=true , address=145.79.227.49:4337 , piece_id=EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.525651226 (915h14m17.396734261s) [noise=true , address=145.79.227.48:4301 , piece_id=HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.536956361 (915h14m17.408039396s) [noise=true , address=145.79.227.16:4909 , piece_id=HS23LZ37LL6QR5D2Q4WFE7W5VQPFCOV2MQNM3PTUCMPCJ2VNDHLQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.624655888 (915h14m17.495738933s) [noise=true , address=45.91.116.6:4974 , piece_id=FRJDOIXXLZMS3FW4X4XAYAIOHTDZ5B7TWCGGM2DY5LTV5RZGU6HQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.629752162 (915h14m17.500835197s) [noise=true , address=45.91.116.7:4012 , piece_id=YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.645478456 (915h14m17.516561491s) [noise=true , address=145.79.227.2:4115 , piece_id=PB2JJEZEXSLTXW5QTHA6IWT32UMAI5EM3S5TL2CGP7A6A3NLA3GA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.676056038 (915h14m17.547139083s) [noise=true , address=45.91.116.14:4189 , piece_id=I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.698909442 (915h14m17.569992488s) [noise=true , address=45.91.116.8:4048 , piece_id=65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.721440313 (915h14m17.592523348s) [noise=true , address=77.90.31.11:4554 , piece_id=H7TROB4ZL5AENH4XYEE3SIFZFXDOE6PKO3AQUA2MACPQYLWJ4XPQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.918612414 (915h14m17.789695459s) [noise=true , address=145.79.227.44:4194 , piece_id=PKZFIIDQYD5O2P25BKIMA45XLXVXZQSLV5H7A2JJGG7XOAHOO2QA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.518838258 (915h14m18.389921293s) [noise=true , address=145.79.227.29:4943 , piece_id=EHLLIAZBTJOBDU2VLA2JXDVQK67WSCRU3FR24HBWK2QF2A5XUERA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.721493959 (915h14m18.592576994s) [noise=true , address=77.90.31.19:4841 , piece_id=PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.727505725 (915h14m18.59858876s) [noise=true , address=77.90.31.12:4590 , piece_id=LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.727530473 (915h14m18.598613518s) [seg_pos={0 0}, upload=801054]: Segment reached optimal threshold of 13 pieces. Jan 12 18:43:18.727812850 (915h14m18.598895885s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.727825525 (915h14m18.59890856s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728214255 (915h14m18.59929729s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.728221819 (915h14m18.599304854s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728386072 (915h14m18.599469107s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.728392494 (915h14m18.599475529s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728413945 (915h14m18.59949698s) [seg_pos={0 0}, upload=801054]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 18:43:18.728512563 (915h14m18.599595598s) [upload=801054]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 18:43:18.728518674 (915h14m18.599601719s) [upload=801054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 18:43:19.041837336 (915h14m18.912920371s) [upload=801054]: Done uploading segments... Jan 12 18:43:19.041856071 (915h14m18.912939116s) [upload=801054]: Upload finished. err: <nil> Duration: 7.238501755s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 32 out of 60 is failed (threshold=6) [X.XX..X.XXXXXX.X.XXX.X.XXXXX.XX....X.XXX.X..X.X.X....XX.....] duration: 3.599210692s trace: https://jaeger.dev.storj.io/trace/64e8accbe0a8a091 FAIL (2026-01-12 18:46:38 UTC) orbiter warning: context deadline exceeded Duration: 3.599241962s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.890238188s trace: https://jaeger.dev.storj.io/trace/de69fa036d8b02 FAIL (2026-01-12 18:46:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ) context deadline exceeded; piecestore: (Node ID: 1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE, Piece ID: EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ) context deadline exceeded; piecestore: (Node ID: 18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29, Piece ID: HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ) context deadline exceeded ======LOGS====== Jan 12 18:43:13.858719809 (915h14m13.729802844s) [upload=801054]: Starting upload Jan 12 18:43:13.858805642 (915h14m13.729888677s) [upload=801054]: Done starting upload Jan 12 18:43:13.858880885 (915h14m13.72996393s) [upload=801054]: Uploading segments... Jan 12 18:43:13.859066038 (915h14m13.730149083s) [upload=801054]: Got next segment. Inline: false Jan 12 18:43:13.859125010 (915h14m13.730208045s) [upload=801054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 18:43:13.859131923 (915h14m13.730214968s) [upload=801054]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 18:43:15.358668982 (915h14m15.229752017s) [seg_pos={0 0}, upload=801054]: Begin upload segment... Jan 12 18:43:15.358714368 (915h14m15.229797413s) [seg_pos={0 0}, upload=801054]: Done begin upload segment. Jan 12 18:43:15.358745317 (915h14m15.229828352s) [noise=true , address=77.90.31.12:4590 , piece_id=LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.358908408 (915h14m15.229991453s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.359387038 (915h14m15.230470083s) [noise=true , address=145.79.227.44:4194 , piece_id=PKZFIIDQYD5O2P25BKIMA45XLXVXZQSLV5H7A2JJGG7XOAHOO2QA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.359851062 (915h14m15.230934107s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360286561 (915h14m15.231369606s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360600609 (915h14m15.231683654s) [noise=true , address=145.79.227.29:4943 , piece_id=EHLLIAZBTJOBDU2VLA2JXDVQK67WSCRU3FR24HBWK2QF2A5XUERA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360877186 (915h14m15.231960231s) [noise=true , address=77.90.31.19:4841 , piece_id=PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.361298538 (915h14m15.232381573s) [noise=true , address=145.79.227.49:4337 , piece_id=EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.362993415 (915h14m15.23407645s) [noise=true , address=45.91.116.14:4189 , piece_id=I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.363095780 (915h14m15.234178825s) [noise=true , address=45.91.116.7:4012 , piece_id=YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.364580467 (915h14m15.235663502s) [noise=true , address=145.79.227.16:4909 , piece_id=HS23LZ37LL6QR5D2Q4WFE7W5VQPFCOV2MQNM3PTUCMPCJ2VNDHLQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.366070103 (915h14m15.237153148s) [noise=true , address=45.91.116.8:4048 , piece_id=65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.367535814 (915h14m15.238618859s) [noise=true , address=145.79.227.48:4301 , piece_id=HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.369525412 (915h14m15.240608457s) [noise=true , address=45.91.116.6:4974 , piece_id=FRJDOIXXLZMS3FW4X4XAYAIOHTDZ5B7TWCGGM2DY5LTV5RZGU6HQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.371063701 (915h14m15.242146736s) [noise=true , address=145.79.227.2:4115 , piece_id=PB2JJEZEXSLTXW5QTHA6IWT32UMAI5EM3S5TL2CGP7A6A3NLA3GA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.372642217 (915h14m15.243725262s) [noise=true , address=77.90.31.11:4554 , piece_id=H7TROB4ZL5AENH4XYEE3SIFZFXDOE6PKO3AQUA2MACPQYLWJ4XPQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:17.184597103 (915h14m17.055680148s) [upload=801054]: Next returned nil segment Jan 12 18:43:17.184643092 (915h14m17.055726127s) [upload=801054]: Waiting for error group managing segments... Jan 12 18:43:17.495324372 (915h14m17.366407407s) [noise=true , address=145.79.227.49:4337 , piece_id=EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.525651226 (915h14m17.396734261s) [noise=true , address=145.79.227.48:4301 , piece_id=HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.536956361 (915h14m17.408039396s) [noise=true , address=145.79.227.16:4909 , piece_id=HS23LZ37LL6QR5D2Q4WFE7W5VQPFCOV2MQNM3PTUCMPCJ2VNDHLQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.624655888 (915h14m17.495738933s) [noise=true , address=45.91.116.6:4974 , piece_id=FRJDOIXXLZMS3FW4X4XAYAIOHTDZ5B7TWCGGM2DY5LTV5RZGU6HQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.629752162 (915h14m17.500835197s) [noise=true , address=45.91.116.7:4012 , piece_id=YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.645478456 (915h14m17.516561491s) [noise=true , address=145.79.227.2:4115 , piece_id=PB2JJEZEXSLTXW5QTHA6IWT32UMAI5EM3S5TL2CGP7A6A3NLA3GA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.676056038 (915h14m17.547139083s) [noise=true , address=45.91.116.14:4189 , piece_id=I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.698909442 (915h14m17.569992488s) [noise=true , address=45.91.116.8:4048 , piece_id=65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.721440313 (915h14m17.592523348s) [noise=true , address=77.90.31.11:4554 , piece_id=H7TROB4ZL5AENH4XYEE3SIFZFXDOE6PKO3AQUA2MACPQYLWJ4XPQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.918612414 (915h14m17.789695459s) [noise=true , address=145.79.227.44:4194 , piece_id=PKZFIIDQYD5O2P25BKIMA45XLXVXZQSLV5H7A2JJGG7XOAHOO2QA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.518838258 (915h14m18.389921293s) [noise=true , address=145.79.227.29:4943 , piece_id=EHLLIAZBTJOBDU2VLA2JXDVQK67WSCRU3FR24HBWK2QF2A5XUERA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.721493959 (915h14m18.592576994s) [noise=true , address=77.90.31.19:4841 , piece_id=PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.727505725 (915h14m18.59858876s) [noise=true , address=77.90.31.12:4590 , piece_id=LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.727530473 (915h14m18.598613518s) [seg_pos={0 0}, upload=801054]: Segment reached optimal threshold of 13 pieces. Jan 12 18:43:18.727812850 (915h14m18.598895885s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.727825525 (915h14m18.59890856s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728214255 (915h14m18.59929729s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.728221819 (915h14m18.599304854s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728386072 (915h14m18.599469107s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.728392494 (915h14m18.599475529s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728413945 (915h14m18.59949698s) [seg_pos={0 0}, upload=801054]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 18:43:18.728512563 (915h14m18.599595598s) [upload=801054]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 18:43:18.728518674 (915h14m18.599601719s) [upload=801054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 18:43:19.041837336 (915h14m18.912920371s) [upload=801054]: Done uploading segments... Jan 12 18:43:19.041856071 (915h14m18.912939116s) [upload=801054]: Upload finished. err: <nil> Duration: 5.89026559s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [XX.XX..X.XXXXXX.X.XXX.X.XXXXX.XX....X.XXX.X..X.X.X....XX....] duration: 3.612115035s trace: https://jaeger.dev.storj.io/trace/64e8accbe0a8a091 FAIL (2026-01-12 18:45:52 UTC) orbiter warning: context deadline exceeded Duration: 3.612130144s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.853996372s trace: https://jaeger.dev.storj.io/trace/de69fa036d8b02 FAIL (2026-01-12 18:45:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ) context deadline exceeded; piecestore: (Node ID: 1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE, Piece ID: EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ) context deadline exceeded; piecestore: (Node ID: 18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29, Piece ID: HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ) context deadline exceeded ======LOGS====== Jan 12 18:43:13.858719809 (915h14m13.729802844s) [upload=801054]: Starting upload Jan 12 18:43:13.858805642 (915h14m13.729888677s) [upload=801054]: Done starting upload Jan 12 18:43:13.858880885 (915h14m13.72996393s) [upload=801054]: Uploading segments... Jan 12 18:43:13.859066038 (915h14m13.730149083s) [upload=801054]: Got next segment. Inline: false Jan 12 18:43:13.859125010 (915h14m13.730208045s) [upload=801054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 18:43:13.859131923 (915h14m13.730214968s) [upload=801054]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 18:43:15.358668982 (915h14m15.229752017s) [seg_pos={0 0}, upload=801054]: Begin upload segment... Jan 12 18:43:15.358714368 (915h14m15.229797413s) [seg_pos={0 0}, upload=801054]: Done begin upload segment. Jan 12 18:43:15.358745317 (915h14m15.229828352s) [noise=true , address=77.90.31.12:4590 , piece_id=LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.358908408 (915h14m15.229991453s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.359387038 (915h14m15.230470083s) [noise=true , address=145.79.227.44:4194 , piece_id=PKZFIIDQYD5O2P25BKIMA45XLXVXZQSLV5H7A2JJGG7XOAHOO2QA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.359851062 (915h14m15.230934107s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360286561 (915h14m15.231369606s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360600609 (915h14m15.231683654s) [noise=true , address=145.79.227.29:4943 , piece_id=EHLLIAZBTJOBDU2VLA2JXDVQK67WSCRU3FR24HBWK2QF2A5XUERA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360877186 (915h14m15.231960231s) [noise=true , address=77.90.31.19:4841 , piece_id=PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.361298538 (915h14m15.232381573s) [noise=true , address=145.79.227.49:4337 , piece_id=EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.362993415 (915h14m15.23407645s) [noise=true , address=45.91.116.14:4189 , piece_id=I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.363095780 (915h14m15.234178825s) [noise=true , address=45.91.116.7:4012 , piece_id=YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.364580467 (915h14m15.235663502s) [noise=true , address=145.79.227.16:4909 , piece_id=HS23LZ37LL6QR5D2Q4WFE7W5VQPFCOV2MQNM3PTUCMPCJ2VNDHLQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.366070103 (915h14m15.237153148s) [noise=true , address=45.91.116.8:4048 , piece_id=65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.367535814 (915h14m15.238618859s) [noise=true , address=145.79.227.48:4301 , piece_id=HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.369525412 (915h14m15.240608457s) [noise=true , address=45.91.116.6:4974 , piece_id=FRJDOIXXLZMS3FW4X4XAYAIOHTDZ5B7TWCGGM2DY5LTV5RZGU6HQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.371063701 (915h14m15.242146736s) [noise=true , address=145.79.227.2:4115 , piece_id=PB2JJEZEXSLTXW5QTHA6IWT32UMAI5EM3S5TL2CGP7A6A3NLA3GA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.372642217 (915h14m15.243725262s) [noise=true , address=77.90.31.11:4554 , piece_id=H7TROB4ZL5AENH4XYEE3SIFZFXDOE6PKO3AQUA2MACPQYLWJ4XPQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:17.184597103 (915h14m17.055680148s) [upload=801054]: Next returned nil segment Jan 12 18:43:17.184643092 (915h14m17.055726127s) [upload=801054]: Waiting for error group managing segments... Jan 12 18:43:17.495324372 (915h14m17.366407407s) [noise=true , address=145.79.227.49:4337 , piece_id=EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.525651226 (915h14m17.396734261s) [noise=true , address=145.79.227.48:4301 , piece_id=HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.536956361 (915h14m17.408039396s) [noise=true , address=145.79.227.16:4909 , piece_id=HS23LZ37LL6QR5D2Q4WFE7W5VQPFCOV2MQNM3PTUCMPCJ2VNDHLQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.624655888 (915h14m17.495738933s) [noise=true , address=45.91.116.6:4974 , piece_id=FRJDOIXXLZMS3FW4X4XAYAIOHTDZ5B7TWCGGM2DY5LTV5RZGU6HQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.629752162 (915h14m17.500835197s) [noise=true , address=45.91.116.7:4012 , piece_id=YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.645478456 (915h14m17.516561491s) [noise=true , address=145.79.227.2:4115 , piece_id=PB2JJEZEXSLTXW5QTHA6IWT32UMAI5EM3S5TL2CGP7A6A3NLA3GA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.676056038 (915h14m17.547139083s) [noise=true , address=45.91.116.14:4189 , piece_id=I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.698909442 (915h14m17.569992488s) [noise=true , address=45.91.116.8:4048 , piece_id=65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.721440313 (915h14m17.592523348s) [noise=true , address=77.90.31.11:4554 , piece_id=H7TROB4ZL5AENH4XYEE3SIFZFXDOE6PKO3AQUA2MACPQYLWJ4XPQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.918612414 (915h14m17.789695459s) [noise=true , address=145.79.227.44:4194 , piece_id=PKZFIIDQYD5O2P25BKIMA45XLXVXZQSLV5H7A2JJGG7XOAHOO2QA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.518838258 (915h14m18.389921293s) [noise=true , address=145.79.227.29:4943 , piece_id=EHLLIAZBTJOBDU2VLA2JXDVQK67WSCRU3FR24HBWK2QF2A5XUERA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.721493959 (915h14m18.592576994s) [noise=true , address=77.90.31.19:4841 , piece_id=PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.727505725 (915h14m18.59858876s) [noise=true , address=77.90.31.12:4590 , piece_id=LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.727530473 (915h14m18.598613518s) [seg_pos={0 0}, upload=801054]: Segment reached optimal threshold of 13 pieces. Jan 12 18:43:18.727812850 (915h14m18.598895885s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.727825525 (915h14m18.59890856s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728214255 (915h14m18.59929729s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.728221819 (915h14m18.599304854s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728386072 (915h14m18.599469107s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.728392494 (915h14m18.599475529s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728413945 (915h14m18.59949698s) [seg_pos={0 0}, upload=801054]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 18:43:18.728512563 (915h14m18.599595598s) [upload=801054]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 18:43:18.728518674 (915h14m18.599601719s) [upload=801054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 18:43:19.041837336 (915h14m18.912920371s) [upload=801054]: Done uploading segments... Jan 12 18:43:19.041856071 (915h14m18.912939116s) [upload=801054]: Upload finished. err: <nil> Duration: 5.854021982s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 33 out of 60 is failed (threshold=6) [.XX.XX..X.XXXXXX.X.XXX.X.XXXXX.XX....X.XXX.X..X.X.X....XX...] duration: 3.869325584s trace: https://jaeger.dev.storj.io/trace/64e8accbe0a8a091 FAIL (2026-01-12 18:45:06 UTC) orbiter warning: context deadline exceeded Duration: 3.869425404s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.937828636s trace: https://jaeger.dev.storj.io/trace/de69fa036d8b02 FAIL (2026-01-12 18:45:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ) context deadline exceeded; piecestore: (Node ID: 1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE, Piece ID: EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ) context deadline exceeded; piecestore: (Node ID: 18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29, Piece ID: HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ) context deadline exceeded ======LOGS====== Jan 12 18:43:13.858719809 (915h14m13.729802844s) [upload=801054]: Starting upload Jan 12 18:43:13.858805642 (915h14m13.729888677s) [upload=801054]: Done starting upload Jan 12 18:43:13.858880885 (915h14m13.72996393s) [upload=801054]: Uploading segments... Jan 12 18:43:13.859066038 (915h14m13.730149083s) [upload=801054]: Got next segment. Inline: false Jan 12 18:43:13.859125010 (915h14m13.730208045s) [upload=801054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 18:43:13.859131923 (915h14m13.730214968s) [upload=801054]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 18:43:15.358668982 (915h14m15.229752017s) [seg_pos={0 0}, upload=801054]: Begin upload segment... Jan 12 18:43:15.358714368 (915h14m15.229797413s) [seg_pos={0 0}, upload=801054]: Done begin upload segment. Jan 12 18:43:15.358745317 (915h14m15.229828352s) [noise=true , address=77.90.31.12:4590 , piece_id=LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.358908408 (915h14m15.229991453s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.359387038 (915h14m15.230470083s) [noise=true , address=145.79.227.44:4194 , piece_id=PKZFIIDQYD5O2P25BKIMA45XLXVXZQSLV5H7A2JJGG7XOAHOO2QA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.359851062 (915h14m15.230934107s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360286561 (915h14m15.231369606s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360600609 (915h14m15.231683654s) [noise=true , address=145.79.227.29:4943 , piece_id=EHLLIAZBTJOBDU2VLA2JXDVQK67WSCRU3FR24HBWK2QF2A5XUERA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360877186 (915h14m15.231960231s) [noise=true , address=77.90.31.19:4841 , piece_id=PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.361298538 (915h14m15.232381573s) [noise=true , address=145.79.227.49:4337 , piece_id=EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.362993415 (915h14m15.23407645s) [noise=true , address=45.91.116.14:4189 , piece_id=I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.363095780 (915h14m15.234178825s) [noise=true , address=45.91.116.7:4012 , piece_id=YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.364580467 (915h14m15.235663502s) [noise=true , address=145.79.227.16:4909 , piece_id=HS23LZ37LL6QR5D2Q4WFE7W5VQPFCOV2MQNM3PTUCMPCJ2VNDHLQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.366070103 (915h14m15.237153148s) [noise=true , address=45.91.116.8:4048 , piece_id=65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.367535814 (915h14m15.238618859s) [noise=true , address=145.79.227.48:4301 , piece_id=HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.369525412 (915h14m15.240608457s) [noise=true , address=45.91.116.6:4974 , piece_id=FRJDOIXXLZMS3FW4X4XAYAIOHTDZ5B7TWCGGM2DY5LTV5RZGU6HQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.371063701 (915h14m15.242146736s) [noise=true , address=145.79.227.2:4115 , piece_id=PB2JJEZEXSLTXW5QTHA6IWT32UMAI5EM3S5TL2CGP7A6A3NLA3GA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.372642217 (915h14m15.243725262s) [noise=true , address=77.90.31.11:4554 , piece_id=H7TROB4ZL5AENH4XYEE3SIFZFXDOE6PKO3AQUA2MACPQYLWJ4XPQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:17.184597103 (915h14m17.055680148s) [upload=801054]: Next returned nil segment Jan 12 18:43:17.184643092 (915h14m17.055726127s) [upload=801054]: Waiting for error group managing segments... Jan 12 18:43:17.495324372 (915h14m17.366407407s) [noise=true , address=145.79.227.49:4337 , piece_id=EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.525651226 (915h14m17.396734261s) [noise=true , address=145.79.227.48:4301 , piece_id=HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.536956361 (915h14m17.408039396s) [noise=true , address=145.79.227.16:4909 , piece_id=HS23LZ37LL6QR5D2Q4WFE7W5VQPFCOV2MQNM3PTUCMPCJ2VNDHLQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.624655888 (915h14m17.495738933s) [noise=true , address=45.91.116.6:4974 , piece_id=FRJDOIXXLZMS3FW4X4XAYAIOHTDZ5B7TWCGGM2DY5LTV5RZGU6HQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.629752162 (915h14m17.500835197s) [noise=true , address=45.91.116.7:4012 , piece_id=YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.645478456 (915h14m17.516561491s) [noise=true , address=145.79.227.2:4115 , piece_id=PB2JJEZEXSLTXW5QTHA6IWT32UMAI5EM3S5TL2CGP7A6A3NLA3GA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.676056038 (915h14m17.547139083s) [noise=true , address=45.91.116.14:4189 , piece_id=I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.698909442 (915h14m17.569992488s) [noise=true , address=45.91.116.8:4048 , piece_id=65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.721440313 (915h14m17.592523348s) [noise=true , address=77.90.31.11:4554 , piece_id=H7TROB4ZL5AENH4XYEE3SIFZFXDOE6PKO3AQUA2MACPQYLWJ4XPQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.918612414 (915h14m17.789695459s) [noise=true , address=145.79.227.44:4194 , piece_id=PKZFIIDQYD5O2P25BKIMA45XLXVXZQSLV5H7A2JJGG7XOAHOO2QA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.518838258 (915h14m18.389921293s) [noise=true , address=145.79.227.29:4943 , piece_id=EHLLIAZBTJOBDU2VLA2JXDVQK67WSCRU3FR24HBWK2QF2A5XUERA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.721493959 (915h14m18.592576994s) [noise=true , address=77.90.31.19:4841 , piece_id=PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.727505725 (915h14m18.59858876s) [noise=true , address=77.90.31.12:4590 , piece_id=LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.727530473 (915h14m18.598613518s) [seg_pos={0 0}, upload=801054]: Segment reached optimal threshold of 13 pieces. Jan 12 18:43:18.727812850 (915h14m18.598895885s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.727825525 (915h14m18.59890856s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728214255 (915h14m18.59929729s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.728221819 (915h14m18.599304854s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728386072 (915h14m18.599469107s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.728392494 (915h14m18.599475529s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728413945 (915h14m18.59949698s) [seg_pos={0 0}, upload=801054]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 18:43:18.728512563 (915h14m18.599595598s) [upload=801054]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 18:43:18.728518674 (915h14m18.599601719s) [upload=801054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 18:43:19.041837336 (915h14m18.912920371s) [upload=801054]: Done uploading segments... Jan 12 18:43:19.041856071 (915h14m18.912939116s) [upload=801054]: Upload finished. err: <nil> Duration: 5.937857482s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.00729924s trace: https://jaeger.dev.storj.io/trace/de69fa036d8b02 FAIL (2026-01-12 18:44:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ) context deadline exceeded; piecestore: (Node ID: 1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE, Piece ID: EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ) context deadline exceeded; piecestore: (Node ID: 18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29, Piece ID: HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ) context deadline exceeded ======LOGS====== Jan 12 18:43:13.858719809 (915h14m13.729802844s) [upload=801054]: Starting upload Jan 12 18:43:13.858805642 (915h14m13.729888677s) [upload=801054]: Done starting upload Jan 12 18:43:13.858880885 (915h14m13.72996393s) [upload=801054]: Uploading segments... Jan 12 18:43:13.859066038 (915h14m13.730149083s) [upload=801054]: Got next segment. Inline: false Jan 12 18:43:13.859125010 (915h14m13.730208045s) [upload=801054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 18:43:13.859131923 (915h14m13.730214968s) [upload=801054]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 18:43:15.358668982 (915h14m15.229752017s) [seg_pos={0 0}, upload=801054]: Begin upload segment... Jan 12 18:43:15.358714368 (915h14m15.229797413s) [seg_pos={0 0}, upload=801054]: Done begin upload segment. Jan 12 18:43:15.358745317 (915h14m15.229828352s) [noise=true , address=77.90.31.12:4590 , piece_id=LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.358908408 (915h14m15.229991453s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.359387038 (915h14m15.230470083s) [noise=true , address=145.79.227.44:4194 , piece_id=PKZFIIDQYD5O2P25BKIMA45XLXVXZQSLV5H7A2JJGG7XOAHOO2QA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.359851062 (915h14m15.230934107s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360286561 (915h14m15.231369606s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360600609 (915h14m15.231683654s) [noise=true , address=145.79.227.29:4943 , piece_id=EHLLIAZBTJOBDU2VLA2JXDVQK67WSCRU3FR24HBWK2QF2A5XUERA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360877186 (915h14m15.231960231s) [noise=true , address=77.90.31.19:4841 , piece_id=PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.361298538 (915h14m15.232381573s) [noise=true , address=145.79.227.49:4337 , piece_id=EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.362993415 (915h14m15.23407645s) [noise=true , address=45.91.116.14:4189 , piece_id=I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.363095780 (915h14m15.234178825s) [noise=true , address=45.91.116.7:4012 , piece_id=YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.364580467 (915h14m15.235663502s) [noise=true , address=145.79.227.16:4909 , piece_id=HS23LZ37LL6QR5D2Q4WFE7W5VQPFCOV2MQNM3PTUCMPCJ2VNDHLQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.366070103 (915h14m15.237153148s) [noise=true , address=45.91.116.8:4048 , piece_id=65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.367535814 (915h14m15.238618859s) [noise=true , address=145.79.227.48:4301 , piece_id=HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.369525412 (915h14m15.240608457s) [noise=true , address=45.91.116.6:4974 , piece_id=FRJDOIXXLZMS3FW4X4XAYAIOHTDZ5B7TWCGGM2DY5LTV5RZGU6HQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.371063701 (915h14m15.242146736s) [noise=true , address=145.79.227.2:4115 , piece_id=PB2JJEZEXSLTXW5QTHA6IWT32UMAI5EM3S5TL2CGP7A6A3NLA3GA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.372642217 (915h14m15.243725262s) [noise=true , address=77.90.31.11:4554 , piece_id=H7TROB4ZL5AENH4XYEE3SIFZFXDOE6PKO3AQUA2MACPQYLWJ4XPQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:17.184597103 (915h14m17.055680148s) [upload=801054]: Next returned nil segment Jan 12 18:43:17.184643092 (915h14m17.055726127s) [upload=801054]: Waiting for error group managing segments... Jan 12 18:43:17.495324372 (915h14m17.366407407s) [noise=true , address=145.79.227.49:4337 , piece_id=EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.525651226 (915h14m17.396734261s) [noise=true , address=145.79.227.48:4301 , piece_id=HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.536956361 (915h14m17.408039396s) [noise=true , address=145.79.227.16:4909 , piece_id=HS23LZ37LL6QR5D2Q4WFE7W5VQPFCOV2MQNM3PTUCMPCJ2VNDHLQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.624655888 (915h14m17.495738933s) [noise=true , address=45.91.116.6:4974 , piece_id=FRJDOIXXLZMS3FW4X4XAYAIOHTDZ5B7TWCGGM2DY5LTV5RZGU6HQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.629752162 (915h14m17.500835197s) [noise=true , address=45.91.116.7:4012 , piece_id=YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.645478456 (915h14m17.516561491s) [noise=true , address=145.79.227.2:4115 , piece_id=PB2JJEZEXSLTXW5QTHA6IWT32UMAI5EM3S5TL2CGP7A6A3NLA3GA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.676056038 (915h14m17.547139083s) [noise=true , address=45.91.116.14:4189 , piece_id=I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.698909442 (915h14m17.569992488s) [noise=true , address=45.91.116.8:4048 , piece_id=65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.721440313 (915h14m17.592523348s) [noise=true , address=77.90.31.11:4554 , piece_id=H7TROB4ZL5AENH4XYEE3SIFZFXDOE6PKO3AQUA2MACPQYLWJ4XPQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.918612414 (915h14m17.789695459s) [noise=true , address=145.79.227.44:4194 , piece_id=PKZFIIDQYD5O2P25BKIMA45XLXVXZQSLV5H7A2JJGG7XOAHOO2QA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.518838258 (915h14m18.389921293s) [noise=true , address=145.79.227.29:4943 , piece_id=EHLLIAZBTJOBDU2VLA2JXDVQK67WSCRU3FR24HBWK2QF2A5XUERA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.721493959 (915h14m18.592576994s) [noise=true , address=77.90.31.19:4841 , piece_id=PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.727505725 (915h14m18.59858876s) [noise=true , address=77.90.31.12:4590 , piece_id=LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.727530473 (915h14m18.598613518s) [seg_pos={0 0}, upload=801054]: Segment reached optimal threshold of 13 pieces. Jan 12 18:43:18.727812850 (915h14m18.598895885s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.727825525 (915h14m18.59890856s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728214255 (915h14m18.59929729s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.728221819 (915h14m18.599304854s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728386072 (915h14m18.599469107s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.728392494 (915h14m18.599475529s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728413945 (915h14m18.59949698s) [seg_pos={0 0}, upload=801054]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 18:43:18.728512563 (915h14m18.599595598s) [upload=801054]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 18:43:18.728518674 (915h14m18.599601719s) [upload=801054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 18:43:19.041837336 (915h14m18.912920371s) [upload=801054]: Done uploading segments... Jan 12 18:43:19.041856071 (915h14m18.912939116s) [upload=801054]: Upload finished. err: <nil> Duration: 6.007323357s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 34 out of 60 is failed (threshold=6) [X.XX.XX..X.XXXXXX.X.XXX.X.XXXXX.XX....X.XXX.X..X.X.X....XX..] duration: 5.001244786s trace: https://jaeger.dev.storj.io/trace/64e8accbe0a8a091 FAIL (2026-01-12 18:44:20 UTC) orbiter warning: context deadline exceeded Duration: 5.001268662s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 27.777952452s trace: https://jaeger.dev.storj.io/trace/de69fa036d8b02 FAIL (2026-01-12 18:44:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ) context deadline exceeded; piecestore: (Node ID: 1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE, Piece ID: EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ) context deadline exceeded; piecestore: (Node ID: 18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29, Piece ID: HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ) context deadline exceeded ======LOGS====== Jan 12 18:43:13.858719809 (915h14m13.729802844s) [upload=801054]: Starting upload Jan 12 18:43:13.858805642 (915h14m13.729888677s) [upload=801054]: Done starting upload Jan 12 18:43:13.858880885 (915h14m13.72996393s) [upload=801054]: Uploading segments... Jan 12 18:43:13.859066038 (915h14m13.730149083s) [upload=801054]: Got next segment. Inline: false Jan 12 18:43:13.859125010 (915h14m13.730208045s) [upload=801054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 18:43:13.859131923 (915h14m13.730214968s) [upload=801054]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 18:43:15.358668982 (915h14m15.229752017s) [seg_pos={0 0}, upload=801054]: Begin upload segment... Jan 12 18:43:15.358714368 (915h14m15.229797413s) [seg_pos={0 0}, upload=801054]: Done begin upload segment. Jan 12 18:43:15.358745317 (915h14m15.229828352s) [noise=true , address=77.90.31.12:4590 , piece_id=LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.358908408 (915h14m15.229991453s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.359387038 (915h14m15.230470083s) [noise=true , address=145.79.227.44:4194 , piece_id=PKZFIIDQYD5O2P25BKIMA45XLXVXZQSLV5H7A2JJGG7XOAHOO2QA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.359851062 (915h14m15.230934107s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360286561 (915h14m15.231369606s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360600609 (915h14m15.231683654s) [noise=true , address=145.79.227.29:4943 , piece_id=EHLLIAZBTJOBDU2VLA2JXDVQK67WSCRU3FR24HBWK2QF2A5XUERA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360877186 (915h14m15.231960231s) [noise=true , address=77.90.31.19:4841 , piece_id=PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.361298538 (915h14m15.232381573s) [noise=true , address=145.79.227.49:4337 , piece_id=EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.362993415 (915h14m15.23407645s) [noise=true , address=45.91.116.14:4189 , piece_id=I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.363095780 (915h14m15.234178825s) [noise=true , address=45.91.116.7:4012 , piece_id=YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.364580467 (915h14m15.235663502s) [noise=true , address=145.79.227.16:4909 , piece_id=HS23LZ37LL6QR5D2Q4WFE7W5VQPFCOV2MQNM3PTUCMPCJ2VNDHLQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.366070103 (915h14m15.237153148s) [noise=true , address=45.91.116.8:4048 , piece_id=65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.367535814 (915h14m15.238618859s) [noise=true , address=145.79.227.48:4301 , piece_id=HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.369525412 (915h14m15.240608457s) [noise=true , address=45.91.116.6:4974 , piece_id=FRJDOIXXLZMS3FW4X4XAYAIOHTDZ5B7TWCGGM2DY5LTV5RZGU6HQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.371063701 (915h14m15.242146736s) [noise=true , address=145.79.227.2:4115 , piece_id=PB2JJEZEXSLTXW5QTHA6IWT32UMAI5EM3S5TL2CGP7A6A3NLA3GA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.372642217 (915h14m15.243725262s) [noise=true , address=77.90.31.11:4554 , piece_id=H7TROB4ZL5AENH4XYEE3SIFZFXDOE6PKO3AQUA2MACPQYLWJ4XPQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:17.184597103 (915h14m17.055680148s) [upload=801054]: Next returned nil segment Jan 12 18:43:17.184643092 (915h14m17.055726127s) [upload=801054]: Waiting for error group managing segments... Jan 12 18:43:17.495324372 (915h14m17.366407407s) [noise=true , address=145.79.227.49:4337 , piece_id=EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.525651226 (915h14m17.396734261s) [noise=true , address=145.79.227.48:4301 , piece_id=HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.536956361 (915h14m17.408039396s) [noise=true , address=145.79.227.16:4909 , piece_id=HS23LZ37LL6QR5D2Q4WFE7W5VQPFCOV2MQNM3PTUCMPCJ2VNDHLQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.624655888 (915h14m17.495738933s) [noise=true , address=45.91.116.6:4974 , piece_id=FRJDOIXXLZMS3FW4X4XAYAIOHTDZ5B7TWCGGM2DY5LTV5RZGU6HQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.629752162 (915h14m17.500835197s) [noise=true , address=45.91.116.7:4012 , piece_id=YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.645478456 (915h14m17.516561491s) [noise=true , address=145.79.227.2:4115 , piece_id=PB2JJEZEXSLTXW5QTHA6IWT32UMAI5EM3S5TL2CGP7A6A3NLA3GA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.676056038 (915h14m17.547139083s) [noise=true , address=45.91.116.14:4189 , piece_id=I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.698909442 (915h14m17.569992488s) [noise=true , address=45.91.116.8:4048 , piece_id=65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.721440313 (915h14m17.592523348s) [noise=true , address=77.90.31.11:4554 , piece_id=H7TROB4ZL5AENH4XYEE3SIFZFXDOE6PKO3AQUA2MACPQYLWJ4XPQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.918612414 (915h14m17.789695459s) [noise=true , address=145.79.227.44:4194 , piece_id=PKZFIIDQYD5O2P25BKIMA45XLXVXZQSLV5H7A2JJGG7XOAHOO2QA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.518838258 (915h14m18.389921293s) [noise=true , address=145.79.227.29:4943 , piece_id=EHLLIAZBTJOBDU2VLA2JXDVQK67WSCRU3FR24HBWK2QF2A5XUERA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.721493959 (915h14m18.592576994s) [noise=true , address=77.90.31.19:4841 , piece_id=PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.727505725 (915h14m18.59858876s) [noise=true , address=77.90.31.12:4590 , piece_id=LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.727530473 (915h14m18.598613518s) [seg_pos={0 0}, upload=801054]: Segment reached optimal threshold of 13 pieces. Jan 12 18:43:18.727812850 (915h14m18.598895885s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.727825525 (915h14m18.59890856s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728214255 (915h14m18.59929729s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.728221819 (915h14m18.599304854s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728386072 (915h14m18.599469107s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.728392494 (915h14m18.599475529s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728413945 (915h14m18.59949698s) [seg_pos={0 0}, upload=801054]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 18:43:18.728512563 (915h14m18.599595598s) [upload=801054]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 18:43:18.728518674 (915h14m18.599601719s) [upload=801054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 18:43:19.041837336 (915h14m18.912920371s) [upload=801054]: Done uploading segments... Jan 12 18:43:19.041856071 (915h14m18.912939116s) [upload=801054]: Upload finished. err: <nil> Duration: 27.777978392s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001664703s trace: https://jaeger.dev.storj.io/trace/de69fa036d8b02 FAIL (2026-01-12 18:43:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ) context deadline exceeded; piecestore: (Node ID: 1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE, Piece ID: EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ) context deadline exceeded; piecestore: (Node ID: 18HkaSrdHHnvmeNz1ormfcSBZH8bMUgkX7X3JmGuqkmtBoak29, Piece ID: HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ) context deadline exceeded ======LOGS====== Jan 12 18:43:13.858719809 (915h14m13.729802844s) [upload=801054]: Starting upload Jan 12 18:43:13.858805642 (915h14m13.729888677s) [upload=801054]: Done starting upload Jan 12 18:43:13.858880885 (915h14m13.72996393s) [upload=801054]: Uploading segments... Jan 12 18:43:13.859066038 (915h14m13.730149083s) [upload=801054]: Got next segment. Inline: false Jan 12 18:43:13.859125010 (915h14m13.730208045s) [upload=801054]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 12 18:43:13.859131923 (915h14m13.730214968s) [upload=801054]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 12 18:43:15.358668982 (915h14m15.229752017s) [seg_pos={0 0}, upload=801054]: Begin upload segment... Jan 12 18:43:15.358714368 (915h14m15.229797413s) [seg_pos={0 0}, upload=801054]: Done begin upload segment. Jan 12 18:43:15.358745317 (915h14m15.229828352s) [noise=true , address=77.90.31.12:4590 , piece_id=LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.358908408 (915h14m15.229991453s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.359387038 (915h14m15.230470083s) [noise=true , address=145.79.227.44:4194 , piece_id=PKZFIIDQYD5O2P25BKIMA45XLXVXZQSLV5H7A2JJGG7XOAHOO2QA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.359851062 (915h14m15.230934107s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360286561 (915h14m15.231369606s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360600609 (915h14m15.231683654s) [noise=true , address=145.79.227.29:4943 , piece_id=EHLLIAZBTJOBDU2VLA2JXDVQK67WSCRU3FR24HBWK2QF2A5XUERA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.360877186 (915h14m15.231960231s) [noise=true , address=77.90.31.19:4841 , piece_id=PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.361298538 (915h14m15.232381573s) [noise=true , address=145.79.227.49:4337 , piece_id=EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.362993415 (915h14m15.23407645s) [noise=true , address=45.91.116.14:4189 , piece_id=I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.363095780 (915h14m15.234178825s) [noise=true , address=45.91.116.7:4012 , piece_id=YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.364580467 (915h14m15.235663502s) [noise=true , address=145.79.227.16:4909 , piece_id=HS23LZ37LL6QR5D2Q4WFE7W5VQPFCOV2MQNM3PTUCMPCJ2VNDHLQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.366070103 (915h14m15.237153148s) [noise=true , address=45.91.116.8:4048 , piece_id=65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.367535814 (915h14m15.238618859s) [noise=true , address=145.79.227.48:4301 , piece_id=HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.369525412 (915h14m15.240608457s) [noise=true , address=45.91.116.6:4974 , piece_id=FRJDOIXXLZMS3FW4X4XAYAIOHTDZ5B7TWCGGM2DY5LTV5RZGU6HQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.371063701 (915h14m15.242146736s) [noise=true , address=145.79.227.2:4115 , piece_id=PB2JJEZEXSLTXW5QTHA6IWT32UMAI5EM3S5TL2CGP7A6A3NLA3GA, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:15.372642217 (915h14m15.243725262s) [noise=true , address=77.90.31.11:4554 , piece_id=H7TROB4ZL5AENH4XYEE3SIFZFXDOE6PKO3AQUA2MACPQYLWJ4XPQ, seg_pos={0 0}, upload=801054]: Uploading piece... Jan 12 18:43:17.184597103 (915h14m17.055680148s) [upload=801054]: Next returned nil segment Jan 12 18:43:17.184643092 (915h14m17.055726127s) [upload=801054]: Waiting for error group managing segments... Jan 12 18:43:17.495324372 (915h14m17.366407407s) [noise=true , address=145.79.227.49:4337 , piece_id=EVS5VGPO6WLN3C2VP3UAVYUYHPXLAZYNRZ2GA2PVZUTXXP3H6O3A, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.525651226 (915h14m17.396734261s) [noise=true , address=145.79.227.48:4301 , piece_id=HRTH34C6ZEOCR5MHVP4DVP4WZ67UU3HRZUBONPRYCIXCDRNLEYNQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.536956361 (915h14m17.408039396s) [noise=true , address=145.79.227.16:4909 , piece_id=HS23LZ37LL6QR5D2Q4WFE7W5VQPFCOV2MQNM3PTUCMPCJ2VNDHLQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.624655888 (915h14m17.495738933s) [noise=true , address=45.91.116.6:4974 , piece_id=FRJDOIXXLZMS3FW4X4XAYAIOHTDZ5B7TWCGGM2DY5LTV5RZGU6HQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.629752162 (915h14m17.500835197s) [noise=true , address=45.91.116.7:4012 , piece_id=YV2L3R4VSD5QKHQZKQWONN4P46FJGOO5L6AZEIOYIWVZCGIZTDGQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.645478456 (915h14m17.516561491s) [noise=true , address=145.79.227.2:4115 , piece_id=PB2JJEZEXSLTXW5QTHA6IWT32UMAI5EM3S5TL2CGP7A6A3NLA3GA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.676056038 (915h14m17.547139083s) [noise=true , address=45.91.116.14:4189 , piece_id=I2EFYW6P6U7ZBBWJCJZYBL72LJAT23FOUKGXUIXFEFVEZTFJU6ZQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.698909442 (915h14m17.569992488s) [noise=true , address=45.91.116.8:4048 , piece_id=65GAA4HMN5XYFMGEAUDDVNK23RYPF2QHIWZ77KPNRZZXR3K4YRFQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.721440313 (915h14m17.592523348s) [noise=true , address=77.90.31.11:4554 , piece_id=H7TROB4ZL5AENH4XYEE3SIFZFXDOE6PKO3AQUA2MACPQYLWJ4XPQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:17.918612414 (915h14m17.789695459s) [noise=true , address=145.79.227.44:4194 , piece_id=PKZFIIDQYD5O2P25BKIMA45XLXVXZQSLV5H7A2JJGG7XOAHOO2QA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.518838258 (915h14m18.389921293s) [noise=true , address=145.79.227.29:4943 , piece_id=EHLLIAZBTJOBDU2VLA2JXDVQK67WSCRU3FR24HBWK2QF2A5XUERA, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.721493959 (915h14m18.592576994s) [noise=true , address=77.90.31.19:4841 , piece_id=PT73NOB5J5RN3YYO2FQOI5WAQHF4VVVMOPRZRS4VAZWPAUBH7G3A, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.727505725 (915h14m18.59858876s) [noise=true , address=77.90.31.12:4590 , piece_id=LHCNG3FICWGL4F7CEF5SAJPHKTWDND6GEZZ6PC33SZMJKE465DSQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: <nil> Jan 12 18:43:18.727530473 (915h14m18.598613518s) [seg_pos={0 0}, upload=801054]: Segment reached optimal threshold of 13 pieces. Jan 12 18:43:18.727812850 (915h14m18.598895885s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.727825525 (915h14m18.59890856s) [noise=true , address=45.91.116.5:4300 , piece_id=WEKZBTQJ3JFDFXNKQ6KDSQ3LKCZOSZTGBR5B65N3WTVRKGBL5R2Q, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728214255 (915h14m18.59929729s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.728221819 (915h14m18.599304854s) [noise=true , address=77.90.31.10:4519 , piece_id=ACYFGWKK7E2TIGLTJVMXZYCLFRX44VXZYHHHTHQYUDDC2NFQXFTQ, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728386072 (915h14m18.599469107s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Jan 12 18:43:18.728392494 (915h14m18.599475529s) [noise=true , address=77.90.31.21:4877 , piece_id=F4RTMXI3C3YHKZ3L4LT5R7MDK37GMN3VGSMDJX22TOWDHLLLMFFQ, seg_pos={0 0}, upload=801054]: Long tail cancellation due to optimal threshold Jan 12 18:43:18.728413945 (915h14m18.59949698s) [seg_pos={0 0}, upload=801054]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 12 18:43:18.728512563 (915h14m18.599595598s) [upload=801054]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 12 18:43:18.728518674 (915h14m18.599601719s) [upload=801054]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 12 18:43:19.041837336 (915h14m18.912920371s) [upload=801054]: Done uploading segments... Jan 12 18:43:19.041856071 (915h14m18.912939116s) [upload=801054]: Upload finished. err: <nil> Duration: 30.001685723s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [XX.XX.XX..X.XXXXXX.X.XXX.X.XXXXX.XX....X.XXX.X..X.X.X....XX.] duration: 5.000287747s trace: https://jaeger.dev.storj.io/trace/64e8accbe0a8a091 FAIL (2026-01-12 18:43:06 UTC) orbiter warning: context deadline exceeded Duration: 5.000329095s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [.XX.XX.XX..X.XXXXXX.X.XXX.X.XXXXX.XX....X.XXX.X..X.X.X....XX] duration: 5.00134462s trace: https://jaeger.dev.storj.io/trace/64e8accbe0a8a091 FAIL (2026-01-12 18:42:38 UTC) orbiter warning: context deadline exceeded Duration: 5.001368535s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [X.XX.XX.XX..X.XXXXXX.X.XXX.X.XXXXX.XX....X.XXX.X..X.X.X....X] duration: 5.000684152s trace: https://jaeger.dev.storj.io/trace/2e7e74808d2f5372 FAIL (2026-01-12 18:41:55 UTC) orbiter warning: context deadline exceeded Duration: 5.000723215s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 35 out of 60 is failed (threshold=6) [XX.XX.XX.XX..X.XXXXXX.X.XXX.X.XXXXX.XX....X.XXX.X..X.X.X....] duration: 3.901477541s trace: https://jaeger.dev.storj.io/trace/1ae2b435195e3f1 FAIL (2026-01-12 18:41:32 UTC) orbiter warning: context deadline exceeded Duration: 3.901505816s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 36 out of 60 is failed (threshold=6) [XXX.XX.XX.XX..X.XXXXXX.X.XXX.X.XXXXX.XX....X.XXX.X..X.X.X...] duration: 5.000643925s trace: https://jaeger.dev.storj.io/trace/1ae2b435195e3f1 FAIL (2026-01-12 18:40:28 UTC) orbiter warning: context deadline exceeded Duration: 5.000671147s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [XXXX.XX.XX.XX..X.XXXXXX.X.XXX.X.XXXXX.XX....X.XXX.X..X.X.X..] duration: 3.549055553s trace: https://jaeger.dev.storj.io/trace/1ae2b435195e3f1 FAIL (2026-01-12 18:39:41 UTC) orbiter warning: context deadline exceeded Duration: 3.54908091s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [.XXXX.XX.XX.XX..X.XXXXXX.X.XXX.X.XXXXX.XX....X.XXX.X..X.X.X.] duration: 4.321392382s trace: https://jaeger.dev.storj.io/trace/1ae2b435195e3f1 FAIL (2026-01-12 18:38:37 UTC) orbiter warning: context deadline exceeded Duration: 4.321417439s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [X.XXXX.XX.XX.XX..X.XXXXXX.X.XXX.X.XXXXX.XX....X.XXX.X..X.X.X] duration: 5.00186891s trace: https://jaeger.dev.storj.io/trace/1ae2b435195e3f1 FAIL (2026-01-12 18:37:34 UTC) orbiter warning: context deadline exceeded Duration: 5.001891273s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 37 out of 60 is failed (threshold=6) [.X.XXXX.XX.XX.XX..X.XXXXXX.X.XXX.X.XXXXX.XX....X.XXX.X..X.X.] duration: 5.000719699s trace: https://jaeger.dev.storj.io/trace/63b71b9f32dfcfce FAIL (2026-01-12 18:37:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000737171s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [X.X.XXXX.XX.XX.XX..X.XXXXXX.X.XXX.X.XXXXX.XX....X.XXX.X..X.X] duration: 5.00167084s trace: https://jaeger.dev.storj.io/trace/63b71b9f32dfcfce FAIL (2026-01-12 18:36:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00169205s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 38 out of 60 is failed (threshold=6) [XX.X.XXXX.XX.XX.XX..X.XXXXXX.X.XXX.X.XXXXX.XX....X.XXX.X..X.] duration: 5.000883954s trace: https://jaeger.dev.storj.io/trace/3e9716ec7c9a2670 FAIL (2026-01-12 18:35:41 UTC) orbiter warning: context deadline exceeded Duration: 5.001028279s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 39 out of 60 is failed (threshold=6) [XXX.X.XXXX.XX.XX.XX..X.XXXXXX.X.XXX.X.XXXXX.XX....X.XXX.X..X] duration: 5.001474246s trace: https://jaeger.dev.storj.io/trace/3e9716ec7c9a2670 FAIL (2026-01-12 18:35:21 UTC) orbiter warning: context deadline exceeded Duration: 5.00149675s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)