Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 906034 reports (Page 11265 of 18121)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 36.65382794s trace: https://jaeger.dev.storj.io/trace/29aa707bf8bd3de6 FAIL (2025-12-30 06:05:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.44:4197 , piece_id=ZNE56BONJR62VVMHXJBHOZZJ5GF3DAIYEW3EEEXUDNEWYDSDDXWQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.517379430 (590h34m25.388462475s) [noise=true , address=193.149.164.143:4695 , piece_id=2HNV6DIKEODV4KEF6LPOUFKCVMXHRV7PUW3DMX7DOQ4F2FXLWUXQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.519724064 (590h34m25.39080711s) [noise=true , address=77.90.31.19:4860 , piece_id=NQPW6LPRAMNTYUXCUX324NJ6FGOIVDL6FLGEX6FA4VXZVP7HOPAQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.522068399 (590h34m25.393151434s) [noise=true , address=145.79.227.49:4347 , piece_id=U6YTLPSWSVAY5PCBDVO4SMJSVZREXSUYF63UHUG5EVGUUEHAMUMQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.523362533 (590h34m25.394445578s) [noise=true , address=64.49.13.169:4706 , piece_id=OLQTYZOJIQIPF3CJQXLGBGLH3U77TDR5SEG7OIRACCGVBZIY2B6Q, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.524404988 (590h34m25.395488023s) [noise=true , address=77.90.31.10:4530 , piece_id=JLPQISS2LHBS4B6OIEAOO6ROUVEALNQNTLR4GIDRZFXIOMKC5P4A, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.525938748 (590h34m25.397021793s) [noise=true , address=145.79.227.35:4487 , piece_id=KIBUZEERFV6L2KIS7DFRUIJU4SX55KMMJNBSTBONPD645RXDTSMA, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.526464530 (590h34m25.397547575s) [noise=true , address=145.79.227.52:4456 , piece_id=MY5TILI3EIVQP3FZV3CPELKGQGKE6473HHVLERGI6FKMNHVXJOGQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.529259672 (590h34m25.400342717s) [noise=true , address=145.79.227.33:4904 , piece_id=QHYAWQQ5RQA4GVBFM6X6FJKMPPKPAIJGHCK3LGBTI5UBZIVWWMBQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.531769822 (590h34m25.402852867s) [noise=true , address=77.90.31.12:4596 , piece_id=C7XX7IICPMICQVXIMOJIVFCW46FMVDXQ3VWBVPJK6OPGRJ3WB4LA, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.532925572 (590h34m25.404008607s) [noise=true , address=145.79.227.40:4064 , piece_id=FZWQQVRADBZQBHEYDGKXWIGKLLFWTFXUFRII42BPGDIQBKCAISMQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.534415720 (590h34m25.405498765s) [noise=true , address=77.90.31.16:4766 , piece_id=LJP5R3IXJAKJ2G3SADNVE3ZO7ZO4LT6ZR67ELMUQMEYMGIRBZQTA, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.537220020 (590h34m25.408303065s) [noise=true , address=145.79.227.3:4183 , piece_id=J46U4OIG35AQEHJI7HUG3K26WDGFWVNWESHG2OPLZ67FEERMQT4Q, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.537490615 (590h34m25.40857365s) [noise=true , address=45.91.116.14:4220 , piece_id=YTRF5POWP5JTYINYW7TZWGVIFONQQBORF5JCARUOJIUSLZ6MX6UA, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.539400482 (590h34m25.410483517s) [noise=true , address=145.79.227.11:4735 , piece_id=7SGLNGJCS325MPNLONOS7USJHUBMAOVR32VQ2CQNH3ESD2CC3QCQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.541128883 (590h34m25.412211918s) [noise=true , address=77.90.31.18:4833 , piece_id=TBQ7TW35NJAREKF2HTVGJW754XOA5QTSDETJPPCFFDU54SIDMMLA, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:26.991260789 (590h34m26.862343834s) [upload=517728]: Next returned nil segment Dec 30 06:03:26.991297679 (590h34m26.862380734s) [upload=517728]: Waiting for error group managing segments... Dec 30 06:03:27.434491232 (590h34m27.305574277s) [noise=true , address=145.79.227.3:4183 , piece_id=J46U4OIG35AQEHJI7HUG3K26WDGFWVNWESHG2OPLZ67FEERMQT4Q, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.473885785 (590h34m27.34496882s) [noise=true , address=145.79.227.11:4735 , piece_id=7SGLNGJCS325MPNLONOS7USJHUBMAOVR32VQ2CQNH3ESD2CC3QCQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.481026933 (590h34m27.352109968s) [noise=true , address=145.79.227.52:4456 , piece_id=MY5TILI3EIVQP3FZV3CPELKGQGKE6473HHVLERGI6FKMNHVXJOGQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.537133078 (590h34m27.408216113s) [noise=true , address=145.79.227.43:4174 , piece_id=PYWCVSCRU45AHKUIGBP6XXOQSR7CS5GIU2W3CV2C5VIAVJAFMYAQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.610086884 (590h34m27.481169919s) [noise=true , address=77.90.31.10:4530 , piece_id=JLPQISS2LHBS4B6OIEAOO6ROUVEALNQNTLR4GIDRZFXIOMKC5P4A, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.682091743 (590h34m27.553174788s) [noise=true , address=77.90.31.12:4596 , piece_id=C7XX7IICPMICQVXIMOJIVFCW46FMVDXQ3VWBVPJK6OPGRJ3WB4LA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.682634627 (590h34m27.553717672s) [noise=true , address=77.90.31.18:4833 , piece_id=TBQ7TW35NJAREKF2HTVGJW754XOA5QTSDETJPPCFFDU54SIDMMLA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.712364247 (590h34m27.583447292s) [noise=true , address=45.91.116.14:4220 , piece_id=YTRF5POWP5JTYINYW7TZWGVIFONQQBORF5JCARUOJIUSLZ6MX6UA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.000168182 (590h34m27.871251227s) [noise=true , address=77.90.31.19:4860 , piece_id=NQPW6LPRAMNTYUXCUX324NJ6FGOIVDL6FLGEX6FA4VXZVP7HOPAQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.289616990 (590h34m28.160700025s) [noise=true , address=145.79.227.44:4197 , piece_id=ZNE56BONJR62VVMHXJBHOZZJ5GF3DAIYEW3EEEXUDNEWYDSDDXWQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.306681834 (590h34m28.177764879s) [noise=true , address=193.149.164.143:4695 , piece_id=2HNV6DIKEODV4KEF6LPOUFKCVMXHRV7PUW3DMX7DOQ4F2FXLWUXQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.312426353 (590h34m28.183509388s) [noise=true , address=145.79.227.30:4779 , piece_id=KBETI2JQ3NDRDFTGFA5FW7S5IYL2TFIZ3NXTKQZX7JZQ5RTMX2LA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.349481942 (590h34m28.220564987s) [noise=true , address=145.79.227.40:4064 , piece_id=FZWQQVRADBZQBHEYDGKXWIGKLLFWTFXUFRII42BPGDIQBKCAISMQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.380570801 (590h34m28.251653836s) [noise=true , address=145.79.227.49:4347 , piece_id=U6YTLPSWSVAY5PCBDVO4SMJSVZREXSUYF63UHUG5EVGUUEHAMUMQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.380591970 (590h34m28.251675016s) [seg_pos={0 0}, upload=517728]: Segment reached optimal threshold of 14 pieces. Dec 30 06:03:28.380822891 (590h34m28.251905926s) [noise=true , address=145.79.227.33:4904 , piece_id=QHYAWQQ5RQA4GVBFM6X6FJKMPPKPAIJGHCK3LGBTI5UBZIVWWMBQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 30 06:03:28.380830565 (590h34m28.2519136s) [noise=true , address=145.79.227.33:4904 , piece_id=QHYAWQQ5RQA4GVBFM6X6FJKMPPKPAIJGHCK3LGBTI5UBZIVWWMBQ, seg_pos={0 0}, upload=517728]: Long tail cancellation due to optimal threshold Dec 30 06:03:28.380925726 (590h34m28.252008771s) [noise=true , address=64.49.13.169:4706 , piece_id=OLQTYZOJIQIPF3CJQXLGBGLH3U77TDR5SEG7OIRACCGVBZIY2B6Q, seg_pos={0 0}, upload=517728]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:55416->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:55416->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55416->64.49.13.169:4706: use of closed network connection Dec 30 06:03:28.380942197 (590h34m28.252025242s) [noise=true , address=64.49.13.169:4706 , piece_id=OLQTYZOJIQIPF3CJQXLGBGLH3U77TDR5SEG7OIRACCGVBZIY2B6Q, seg_pos={0 0}, upload=517728]: Long tail cancellation due to optimal threshold Dec 30 06:03:28.380958769 (590h34m28.252041814s) [noise=true , address=145.79.227.35:4487 , piece_id=KIBUZEERFV6L2KIS7DFRUIJU4SX55KMMJNBSTBONPD645RXDTSMA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AdZLDV7UBbV2DimCHzmoNxycLigvLMx6Fy9U9QB35WTtiV1KX): protocol: expected piece hash; context canceled; EOF Dec 30 06:03:28.380967896 (590h34m28.252050942s) [noise=true , address=145.79.227.35:4487 , piece_id=KIBUZEERFV6L2KIS7DFRUIJU4SX55KMMJNBSTBONPD645RXDTSMA, seg_pos={0 0}, upload=517728]: Long tail cancellation due to optimal threshold Dec 30 06:03:28.381091813 (590h34m28.252174858s) [noise=true , address=77.90.31.16:4766 , piece_id=LJP5R3IXJAKJ2G3SADNVE3ZO7ZO4LT6ZR67ELMUQMEYMGIRBZQTA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK): protocol: expected piece hash; context canceled; EOF Dec 30 06:03:28.381098525 (590h34m28.252181561s) [noise=true , address=77.90.31.16:4766 , piece_id=LJP5R3IXJAKJ2G3SADNVE3ZO7ZO4LT6ZR67ELMUQMEYMGIRBZQTA, seg_pos={0 0}, upload=517728]: Long tail cancellation due to optimal threshold Dec 30 06:03:28.381144553 (590h34m28.252227588s) [seg_pos={0 0}, upload=517728]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 06:03:28.381225898 (590h34m28.252308933s) [upload=517728]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:03:28.381231308 (590h34m28.252314344s) [upload=517728]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:03:28.673388913 (590h34m28.544471958s) [upload=517728]: Done uploading segments... Dec 30 06:03:28.673412569 (590h34m28.544495604s) [upload=517728]: Upload finished. err: <nil> Duration: 36.653850211s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 3.96941717s trace: https://jaeger.dev.storj.io/trace/51fe686d4b8ba83d FAIL (2025-12-30 06:04:47 UTC) orbiter warning: context deadline exceeded Duration: 3.969446636s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.004412389s trace: https://jaeger.dev.storj.io/trace/29aa707bf8bd3de6 FAIL (2025-12-30 06:04:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: context deadline exceeded ======LOGS====== ) [noise=true , address=145.79.227.44:4197 , piece_id=ZNE56BONJR62VVMHXJBHOZZJ5GF3DAIYEW3EEEXUDNEWYDSDDXWQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.517379430 (590h34m25.388462475s) [noise=true , address=193.149.164.143:4695 , piece_id=2HNV6DIKEODV4KEF6LPOUFKCVMXHRV7PUW3DMX7DOQ4F2FXLWUXQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.519724064 (590h34m25.39080711s) [noise=true , address=77.90.31.19:4860 , piece_id=NQPW6LPRAMNTYUXCUX324NJ6FGOIVDL6FLGEX6FA4VXZVP7HOPAQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.522068399 (590h34m25.393151434s) [noise=true , address=145.79.227.49:4347 , piece_id=U6YTLPSWSVAY5PCBDVO4SMJSVZREXSUYF63UHUG5EVGUUEHAMUMQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.523362533 (590h34m25.394445578s) [noise=true , address=64.49.13.169:4706 , piece_id=OLQTYZOJIQIPF3CJQXLGBGLH3U77TDR5SEG7OIRACCGVBZIY2B6Q, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.524404988 (590h34m25.395488023s) [noise=true , address=77.90.31.10:4530 , piece_id=JLPQISS2LHBS4B6OIEAOO6ROUVEALNQNTLR4GIDRZFXIOMKC5P4A, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.525938748 (590h34m25.397021793s) [noise=true , address=145.79.227.35:4487 , piece_id=KIBUZEERFV6L2KIS7DFRUIJU4SX55KMMJNBSTBONPD645RXDTSMA, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.526464530 (590h34m25.397547575s) [noise=true , address=145.79.227.52:4456 , piece_id=MY5TILI3EIVQP3FZV3CPELKGQGKE6473HHVLERGI6FKMNHVXJOGQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.529259672 (590h34m25.400342717s) [noise=true , address=145.79.227.33:4904 , piece_id=QHYAWQQ5RQA4GVBFM6X6FJKMPPKPAIJGHCK3LGBTI5UBZIVWWMBQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.531769822 (590h34m25.402852867s) [noise=true , address=77.90.31.12:4596 , piece_id=C7XX7IICPMICQVXIMOJIVFCW46FMVDXQ3VWBVPJK6OPGRJ3WB4LA, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.532925572 (590h34m25.404008607s) [noise=true , address=145.79.227.40:4064 , piece_id=FZWQQVRADBZQBHEYDGKXWIGKLLFWTFXUFRII42BPGDIQBKCAISMQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.534415720 (590h34m25.405498765s) [noise=true , address=77.90.31.16:4766 , piece_id=LJP5R3IXJAKJ2G3SADNVE3ZO7ZO4LT6ZR67ELMUQMEYMGIRBZQTA, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.537220020 (590h34m25.408303065s) [noise=true , address=145.79.227.3:4183 , piece_id=J46U4OIG35AQEHJI7HUG3K26WDGFWVNWESHG2OPLZ67FEERMQT4Q, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.537490615 (590h34m25.40857365s) [noise=true , address=45.91.116.14:4220 , piece_id=YTRF5POWP5JTYINYW7TZWGVIFONQQBORF5JCARUOJIUSLZ6MX6UA, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.539400482 (590h34m25.410483517s) [noise=true , address=145.79.227.11:4735 , piece_id=7SGLNGJCS325MPNLONOS7USJHUBMAOVR32VQ2CQNH3ESD2CC3QCQ, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:25.541128883 (590h34m25.412211918s) [noise=true , address=77.90.31.18:4833 , piece_id=TBQ7TW35NJAREKF2HTVGJW754XOA5QTSDETJPPCFFDU54SIDMMLA, seg_pos={0 0}, upload=517728]: Uploading piece... Dec 30 06:03:26.991260789 (590h34m26.862343834s) [upload=517728]: Next returned nil segment Dec 30 06:03:26.991297679 (590h34m26.862380734s) [upload=517728]: Waiting for error group managing segments... Dec 30 06:03:27.434491232 (590h34m27.305574277s) [noise=true , address=145.79.227.3:4183 , piece_id=J46U4OIG35AQEHJI7HUG3K26WDGFWVNWESHG2OPLZ67FEERMQT4Q, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.473885785 (590h34m27.34496882s) [noise=true , address=145.79.227.11:4735 , piece_id=7SGLNGJCS325MPNLONOS7USJHUBMAOVR32VQ2CQNH3ESD2CC3QCQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.481026933 (590h34m27.352109968s) [noise=true , address=145.79.227.52:4456 , piece_id=MY5TILI3EIVQP3FZV3CPELKGQGKE6473HHVLERGI6FKMNHVXJOGQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.537133078 (590h34m27.408216113s) [noise=true , address=145.79.227.43:4174 , piece_id=PYWCVSCRU45AHKUIGBP6XXOQSR7CS5GIU2W3CV2C5VIAVJAFMYAQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.610086884 (590h34m27.481169919s) [noise=true , address=77.90.31.10:4530 , piece_id=JLPQISS2LHBS4B6OIEAOO6ROUVEALNQNTLR4GIDRZFXIOMKC5P4A, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.682091743 (590h34m27.553174788s) [noise=true , address=77.90.31.12:4596 , piece_id=C7XX7IICPMICQVXIMOJIVFCW46FMVDXQ3VWBVPJK6OPGRJ3WB4LA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.682634627 (590h34m27.553717672s) [noise=true , address=77.90.31.18:4833 , piece_id=TBQ7TW35NJAREKF2HTVGJW754XOA5QTSDETJPPCFFDU54SIDMMLA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:27.712364247 (590h34m27.583447292s) [noise=true , address=45.91.116.14:4220 , piece_id=YTRF5POWP5JTYINYW7TZWGVIFONQQBORF5JCARUOJIUSLZ6MX6UA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.000168182 (590h34m27.871251227s) [noise=true , address=77.90.31.19:4860 , piece_id=NQPW6LPRAMNTYUXCUX324NJ6FGOIVDL6FLGEX6FA4VXZVP7HOPAQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.289616990 (590h34m28.160700025s) [noise=true , address=145.79.227.44:4197 , piece_id=ZNE56BONJR62VVMHXJBHOZZJ5GF3DAIYEW3EEEXUDNEWYDSDDXWQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.306681834 (590h34m28.177764879s) [noise=true , address=193.149.164.143:4695 , piece_id=2HNV6DIKEODV4KEF6LPOUFKCVMXHRV7PUW3DMX7DOQ4F2FXLWUXQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.312426353 (590h34m28.183509388s) [noise=true , address=145.79.227.30:4779 , piece_id=KBETI2JQ3NDRDFTGFA5FW7S5IYL2TFIZ3NXTKQZX7JZQ5RTMX2LA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.349481942 (590h34m28.220564987s) [noise=true , address=145.79.227.40:4064 , piece_id=FZWQQVRADBZQBHEYDGKXWIGKLLFWTFXUFRII42BPGDIQBKCAISMQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.380570801 (590h34m28.251653836s) [noise=true , address=145.79.227.49:4347 , piece_id=U6YTLPSWSVAY5PCBDVO4SMJSVZREXSUYF63UHUG5EVGUUEHAMUMQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: <nil> Dec 30 06:03:28.380591970 (590h34m28.251675016s) [seg_pos={0 0}, upload=517728]: Segment reached optimal threshold of 14 pieces. Dec 30 06:03:28.380822891 (590h34m28.251905926s) [noise=true , address=145.79.227.33:4904 , piece_id=QHYAWQQ5RQA4GVBFM6X6FJKMPPKPAIJGHCK3LGBTI5UBZIVWWMBQ, seg_pos={0 0}, upload=517728]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC): protocol: expected piece hash; context canceled; EOF Dec 30 06:03:28.380830565 (590h34m28.2519136s) [noise=true , address=145.79.227.33:4904 , piece_id=QHYAWQQ5RQA4GVBFM6X6FJKMPPKPAIJGHCK3LGBTI5UBZIVWWMBQ, seg_pos={0 0}, upload=517728]: Long tail cancellation due to optimal threshold Dec 30 06:03:28.380925726 (590h34m28.252008771s) [noise=true , address=64.49.13.169:4706 , piece_id=OLQTYZOJIQIPF3CJQXLGBGLH3U77TDR5SEG7OIRACCGVBZIY2B6Q, seg_pos={0 0}, upload=517728]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xwktmeUvkUf2wfqQTazBsknPs2aC7sj1GLqR875qcW4Cj2GD7): protocol: write tcp 188.166.225.170:55416->64.49.13.169:4706: use of closed network connection; write tcp 188.166.225.170:55416->64.49.13.169:4706: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55416->64.49.13.169:4706: use of closed network connection Dec 30 06:03:28.380942197 (590h34m28.252025242s) [noise=true , address=64.49.13.169:4706 , piece_id=OLQTYZOJIQIPF3CJQXLGBGLH3U77TDR5SEG7OIRACCGVBZIY2B6Q, seg_pos={0 0}, upload=517728]: Long tail cancellation due to optimal threshold Dec 30 06:03:28.380958769 (590h34m28.252041814s) [noise=true , address=145.79.227.35:4487 , piece_id=KIBUZEERFV6L2KIS7DFRUIJU4SX55KMMJNBSTBONPD645RXDTSMA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AdZLDV7UBbV2DimCHzmoNxycLigvLMx6Fy9U9QB35WTtiV1KX): protocol: expected piece hash; context canceled; EOF Dec 30 06:03:28.380967896 (590h34m28.252050942s) [noise=true , address=145.79.227.35:4487 , piece_id=KIBUZEERFV6L2KIS7DFRUIJU4SX55KMMJNBSTBONPD645RXDTSMA, seg_pos={0 0}, upload=517728]: Long tail cancellation due to optimal threshold Dec 30 06:03:28.381091813 (590h34m28.252174858s) [noise=true , address=77.90.31.16:4766 , piece_id=LJP5R3IXJAKJ2G3SADNVE3ZO7ZO4LT6ZR67ELMUQMEYMGIRBZQTA, seg_pos={0 0}, upload=517728]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1V8iMvJLcTddWZihWQBQLLCyMTwbD2h7ruDUjMEtHD7EHtNVEK): protocol: expected piece hash; context canceled; EOF Dec 30 06:03:28.381098525 (590h34m28.252181561s) [noise=true , address=77.90.31.16:4766 , piece_id=LJP5R3IXJAKJ2G3SADNVE3ZO7ZO4LT6ZR67ELMUQMEYMGIRBZQTA, seg_pos={0 0}, upload=517728]: Long tail cancellation due to optimal threshold Dec 30 06:03:28.381144553 (590h34m28.252227588s) [seg_pos={0 0}, upload=517728]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 06:03:28.381225898 (590h34m28.252308933s) [upload=517728]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:03:28.381231308 (590h34m28.252314344s) [upload=517728]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:03:28.673388913 (590h34m28.544471958s) [upload=517728]: Done uploading segments... Dec 30 06:03:28.673412569 (590h34m28.544495604s) [upload=517728]: Upload finished. err: <nil> Duration: 1m0.004427459s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X...X....X.X.......X..X..X....X.XX.X..X..X..X..XX.......X.] duration: 5.768885468s trace: https://jaeger.dev.storj.io/trace/1237689ed7bb48cb FAIL (2025-12-30 06:04:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: JDKZBALSEX6P4YGMWJWAI6ISCWC7DU5RKWCIJGIHK4YCTQOKMFOA) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: ECNNYY6OAOBKZ55GGIX5E5YGMQKXUA6H4OPBMR3M32FYBO3NBXQA) context deadline exceeded ======LOGS====== Dec 30 06:03:43.669744944 (590h34m43.540827989s) [upload=517732]: Starting upload Dec 30 06:03:43.669856857 (590h34m43.540939902s) [upload=517732]: Done starting upload Dec 30 06:03:43.670047460 (590h34m43.541130515s) [upload=517732]: Uploading segments... Dec 30 06:03:43.670118325 (590h34m43.54120138s) [upload=517732]: Got next segment. Inline: false Dec 30 06:03:43.670156988 (590h34m43.541240063s) [upload=517732]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 06:03:43.670163641 (590h34m43.541246686s) [upload=517732]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 06:03:43.917142178 (590h34m43.788225213s) [seg_pos={0 0}, upload=517732]: Begin upload segment... Dec 30 06:03:43.917189789 (590h34m43.788272834s) [seg_pos={0 0}, upload=517732]: Done begin upload segment. Dec 30 06:03:43.917215047 (590h34m43.788298082s) [noise=true , address=145.79.227.14:4837 , piece_id=2RR2YOJE4Z2HPO662QZMAH4BWUJIVP26T3LUHM4N6Y3CQEPWDB7Q, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.918021964 (590h34m43.789105019s) [noise=true , address=145.79.227.15:4873 , piece_id=G2V6NLM4Q2JGZOB7CIX224JYFQUXPCQXR6YR3YZ7HU7QNP7LPQRA, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.919021606 (590h34m43.790104651s) [noise=true , address=77.90.31.14:4664 , piece_id=V6BZMBCU5Z6ZKUXNZ7E7AGRTKUNTMDEA3532BVP5K2DJKRTDXK6Q, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.919502883 (590h34m43.790585928s) [noise=true , address=145.79.227.45:4229 , piece_id=AZBFGM6B4B6XAKIJMEGT3NY542RPI4GYGXQOHRMHN4MBSWIH7HZQ, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.919976906 (590h34m43.791059951s) [noise=true , address=145.79.227.10:4402 , piece_id=RB4ELKAZWOZYJFPVJSSQSQ4A6MQ3QABD64ZOFBMBGBLP4ZBLYTCQ, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.920435189 (590h34m43.791518234s) [noise=true , address=45.91.116.10:4084 , piece_id=SLRPH763LLXQQN4W7NO3W4UOTGXICMTBZC345F7GXKO5IBNZTOYA, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.921350631 (590h34m43.792433666s) [noise=true , address=145.79.227.12:4765 , piece_id=IFHTZWQASQV57XXQ4CK4OBDCMCZLFNWA6MK7B3J5SOGDUGKLQ75A, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.922041407 (590h34m43.793124452s) [noise=true , address=45.91.116.5:4297 , piece_id=BC76YOSI7JD6KIC6O2ZIH5EAEQSRUI5TQ33RKEC53PAQA7Q2OLIQ, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.922881406 (590h34m43.793964461s) [noise=true , address=77.90.31.18:4805 , piece_id=JDKZBALSEX6P4YGMWJWAI6ISCWC7DU5RKWCIJGIHK4YCTQOKMFOA, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.923811317 (590h34m43.794894372s) [noise=true , address=45.91.116.7:4012 , piece_id=ECNNYY6OAOBKZ55GGIX5E5YGMQKXUA6H4OPBMR3M32FYBO3NBXQA, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.924751988 (590h34m43.795835023s) [noise=true , address=45.91.116.14:4190 , piece_id=5Q4LQPWJYNJTZA6KYOIXOHBYEJ3KZBYTMFC6DAV637PVJBMMJGCA, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.925483261 (590h34m43.796566306s) [noise=true , address=145.79.227.8:4329 , piece_id=2U24RSR6RO4AUNPUKIL2EHPXFIIJV6J54Q5XV37Z7DUJ47YVLUOA, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.926350623 (590h34m43.797433658s) [noise=true , address=145.79.227.11:4731 , piece_id=TIIT6OYWB5WSC2YW44KYDOVM4RLZXFAQDSDVKSRBCBDYJ5XRZRNA, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.927963083 (590h34m43.799046118s) [noise=true , address=145.79.227.38:4978 , piece_id=TCCP6WCVDGLV7DRNMZLZFO5FTGLIV2CWEUEOWJ7LW3GJMMASPZMQ, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.929085090 (590h34m43.800168135s) [noise=true , address=45.91.116.12:4225 , piece_id=WYVRJN2ERQ47GJHFNIX27A3VAPZH7UL3BH4YGFXU5MBCCW4DUVYQ, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.930062000 (590h34m43.801145045s) [noise=true , address=145.79.227.5:4223 , piece_id=M52N3HJ37KOPA3LNGSIYM5V57YE4ELOONXUMD54JNIUJLMYGHZHQ, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:45.617481040 (590h34m45.488564085s) [upload=517732]: Next returned nil segment Dec 30 06:03:45.617511608 (590h34m45.488594654s) [upload=517732]: Waiting for error group managing segments... Dec 30 06:03:45.903336394 (590h34m45.774419439s) [noise=true , address=145.79.227.45:4229 , piece_id=AZBFGM6B4B6XAKIJMEGT3NY542RPI4GYGXQOHRMHN4MBSWIH7HZQ, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:45.910012327 (590h34m45.781095372s) [noise=true , address=145.79.227.5:4223 , piece_id=M52N3HJ37KOPA3LNGSIYM5V57YE4ELOONXUMD54JNIUJLMYGHZHQ, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:45.914200573 (590h34m45.785283618s) [noise=true , address=145.79.227.10:4402 , piece_id=RB4ELKAZWOZYJFPVJSSQSQ4A6MQ3QABD64ZOFBMBGBLP4ZBLYTCQ, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:45.930777117 (590h34m45.801860162s) [noise=true , address=145.79.227.8:4329 , piece_id=2U24RSR6RO4AUNPUKIL2EHPXFIIJV6J54Q5XV37Z7DUJ47YVLUOA, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:45.931789354 (590h34m45.802872399s) [noise=true , address=145.79.227.15:4873 , piece_id=G2V6NLM4Q2JGZOB7CIX224JYFQUXPCQXR6YR3YZ7HU7QNP7LPQRA, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:45.933085292 (590h34m45.804168337s) [noise=true , address=145.79.227.14:4837 , piece_id=2RR2YOJE4Z2HPO662QZMAH4BWUJIVP26T3LUHM4N6Y3CQEPWDB7Q, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:45.945017462 (590h34m45.816100497s) [noise=true , address=145.79.227.12:4765 , piece_id=IFHTZWQASQV57XXQ4CK4OBDCMCZLFNWA6MK7B3J5SOGDUGKLQ75A, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:45.997781494 (590h34m45.868864539s) [noise=true , address=145.79.227.11:4731 , piece_id=TIIT6OYWB5WSC2YW44KYDOVM4RLZXFAQDSDVKSRBCBDYJ5XRZRNA, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:46.006849841 (590h34m45.877932876s) [noise=true , address=145.79.227.38:4978 , piece_id=TCCP6WCVDGLV7DRNMZLZFO5FTGLIV2CWEUEOWJ7LW3GJMMASPZMQ, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:46.225565652 (590h34m46.096648687s) [noise=true , address=77.90.31.14:4664 , piece_id=V6BZMBCU5Z6ZKUXNZ7E7AGRTKUNTMDEA3532BVP5K2DJKRTDXK6Q, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:46.265443075 (590h34m46.13652612s) [noise=true , address=45.91.116.5:4297 , piece_id=BC76YOSI7JD6KIC6O2ZIH5EAEQSRUI5TQ33RKEC53PAQA7Q2OLIQ, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:46.600316007 (590h34m46.471399052s) [noise=true , address=45.91.116.7:4012 , piece_id=ECNNYY6OAOBKZ55GGIX5E5YGMQKXUA6H4OPBMR3M32FYBO3NBXQA, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:46.801841444 (590h34m46.672924519s) [noise=true , address=77.90.31.18:4805 , piece_id=JDKZBALSEX6P4YGMWJWAI6ISCWC7DU5RKWCIJGIHK4YCTQOKMFOA, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:46.801867023 (590h34m46.672950058s) [seg_pos={0 0}, upload=517732]: Segment reached optimal threshold of 13 pieces. Dec 30 06:03:46.802099957 (590h34m46.673183002s) [noise=true , address=45.91.116.10:4084 , piece_id=SLRPH763LLXQQN4W7NO3W4UOTGXICMTBZC345F7GXKO5IBNZTOYA, seg_pos={0 0}, upload=517732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 30 06:03:46.802110587 (590h34m46.673193622s) [noise=true , address=45.91.116.10:4084 , piece_id=SLRPH763LLXQQN4W7NO3W4UOTGXICMTBZC345F7GXKO5IBNZTOYA, seg_pos={0 0}, upload=517732]: Long tail cancellation due to optimal threshold Dec 30 06:03:46.802131887 (590h34m46.673214932s) [noise=true , address=45.91.116.12:4225 , piece_id=WYVRJN2ERQ47GJHFNIX27A3VAPZH7UL3BH4YGFXU5MBCCW4DUVYQ, seg_pos={0 0}, upload=517732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 30 06:03:46.802139983 (590h34m46.673223038s) [noise=true , address=45.91.116.12:4225 , piece_id=WYVRJN2ERQ47GJHFNIX27A3VAPZH7UL3BH4YGFXU5MBCCW4DUVYQ, seg_pos={0 0}, upload=517732]: Long tail cancellation due to optimal threshold Dec 30 06:03:46.802472696 (590h34m46.673555741s) [noise=true , address=45.91.116.14:4190 , piece_id=5Q4LQPWJYNJTZA6KYOIXOHBYEJ3KZBYTMFC6DAV637PVJBMMJGCA, seg_pos={0 0}, upload=517732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 30 06:03:46.802483287 (590h34m46.673566322s) [noise=true , address=45.91.116.14:4190 , piece_id=5Q4LQPWJYNJTZA6KYOIXOHBYEJ3KZBYTMFC6DAV637PVJBMMJGCA, seg_pos={0 0}, upload=517732]: Long tail cancellation due to optimal threshold Dec 30 06:03:46.802509205 (590h34m46.67359224s) [seg_pos={0 0}, upload=517732]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 06:03:46.802602804 (590h34m46.673685859s) [upload=517732]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:03:46.802610218 (590h34m46.673693263s) [upload=517732]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:03:47.099426405 (590h34m46.97050946s) [upload=517732]: Done uploading segments... Dec 30 06:03:47.099449608 (590h34m46.970532653s) [upload=517732]: Upload finished. err: <nil> Duration: 5.76890227s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X..X...X....X.X.......X..X..X....X.XX.X..X..X..X..XX.......X] duration: 30.003401378s trace: https://jaeger.dev.storj.io/trace/1237689ed7bb48cb FAIL (2025-12-30 06:04:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: JDKZBALSEX6P4YGMWJWAI6ISCWC7DU5RKWCIJGIHK4YCTQOKMFOA) context deadline exceeded; piecestore: (Node ID: 1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, Piece ID: ECNNYY6OAOBKZ55GGIX5E5YGMQKXUA6H4OPBMR3M32FYBO3NBXQA) context deadline exceeded ======LOGS====== Dec 30 06:03:43.669744944 (590h34m43.540827989s) [upload=517732]: Starting upload Dec 30 06:03:43.669856857 (590h34m43.540939902s) [upload=517732]: Done starting upload Dec 30 06:03:43.670047460 (590h34m43.541130515s) [upload=517732]: Uploading segments... Dec 30 06:03:43.670118325 (590h34m43.54120138s) [upload=517732]: Got next segment. Inline: false Dec 30 06:03:43.670156988 (590h34m43.541240063s) [upload=517732]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 06:03:43.670163641 (590h34m43.541246686s) [upload=517732]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 06:03:43.917142178 (590h34m43.788225213s) [seg_pos={0 0}, upload=517732]: Begin upload segment... Dec 30 06:03:43.917189789 (590h34m43.788272834s) [seg_pos={0 0}, upload=517732]: Done begin upload segment. Dec 30 06:03:43.917215047 (590h34m43.788298082s) [noise=true , address=145.79.227.14:4837 , piece_id=2RR2YOJE4Z2HPO662QZMAH4BWUJIVP26T3LUHM4N6Y3CQEPWDB7Q, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.918021964 (590h34m43.789105019s) [noise=true , address=145.79.227.15:4873 , piece_id=G2V6NLM4Q2JGZOB7CIX224JYFQUXPCQXR6YR3YZ7HU7QNP7LPQRA, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.919021606 (590h34m43.790104651s) [noise=true , address=77.90.31.14:4664 , piece_id=V6BZMBCU5Z6ZKUXNZ7E7AGRTKUNTMDEA3532BVP5K2DJKRTDXK6Q, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.919502883 (590h34m43.790585928s) [noise=true , address=145.79.227.45:4229 , piece_id=AZBFGM6B4B6XAKIJMEGT3NY542RPI4GYGXQOHRMHN4MBSWIH7HZQ, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.919976906 (590h34m43.791059951s) [noise=true , address=145.79.227.10:4402 , piece_id=RB4ELKAZWOZYJFPVJSSQSQ4A6MQ3QABD64ZOFBMBGBLP4ZBLYTCQ, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.920435189 (590h34m43.791518234s) [noise=true , address=45.91.116.10:4084 , piece_id=SLRPH763LLXQQN4W7NO3W4UOTGXICMTBZC345F7GXKO5IBNZTOYA, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.921350631 (590h34m43.792433666s) [noise=true , address=145.79.227.12:4765 , piece_id=IFHTZWQASQV57XXQ4CK4OBDCMCZLFNWA6MK7B3J5SOGDUGKLQ75A, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.922041407 (590h34m43.793124452s) [noise=true , address=45.91.116.5:4297 , piece_id=BC76YOSI7JD6KIC6O2ZIH5EAEQSRUI5TQ33RKEC53PAQA7Q2OLIQ, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.922881406 (590h34m43.793964461s) [noise=true , address=77.90.31.18:4805 , piece_id=JDKZBALSEX6P4YGMWJWAI6ISCWC7DU5RKWCIJGIHK4YCTQOKMFOA, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.923811317 (590h34m43.794894372s) [noise=true , address=45.91.116.7:4012 , piece_id=ECNNYY6OAOBKZ55GGIX5E5YGMQKXUA6H4OPBMR3M32FYBO3NBXQA, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.924751988 (590h34m43.795835023s) [noise=true , address=45.91.116.14:4190 , piece_id=5Q4LQPWJYNJTZA6KYOIXOHBYEJ3KZBYTMFC6DAV637PVJBMMJGCA, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.925483261 (590h34m43.796566306s) [noise=true , address=145.79.227.8:4329 , piece_id=2U24RSR6RO4AUNPUKIL2EHPXFIIJV6J54Q5XV37Z7DUJ47YVLUOA, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.926350623 (590h34m43.797433658s) [noise=true , address=145.79.227.11:4731 , piece_id=TIIT6OYWB5WSC2YW44KYDOVM4RLZXFAQDSDVKSRBCBDYJ5XRZRNA, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.927963083 (590h34m43.799046118s) [noise=true , address=145.79.227.38:4978 , piece_id=TCCP6WCVDGLV7DRNMZLZFO5FTGLIV2CWEUEOWJ7LW3GJMMASPZMQ, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.929085090 (590h34m43.800168135s) [noise=true , address=45.91.116.12:4225 , piece_id=WYVRJN2ERQ47GJHFNIX27A3VAPZH7UL3BH4YGFXU5MBCCW4DUVYQ, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:43.930062000 (590h34m43.801145045s) [noise=true , address=145.79.227.5:4223 , piece_id=M52N3HJ37KOPA3LNGSIYM5V57YE4ELOONXUMD54JNIUJLMYGHZHQ, seg_pos={0 0}, upload=517732]: Uploading piece... Dec 30 06:03:45.617481040 (590h34m45.488564085s) [upload=517732]: Next returned nil segment Dec 30 06:03:45.617511608 (590h34m45.488594654s) [upload=517732]: Waiting for error group managing segments... Dec 30 06:03:45.903336394 (590h34m45.774419439s) [noise=true , address=145.79.227.45:4229 , piece_id=AZBFGM6B4B6XAKIJMEGT3NY542RPI4GYGXQOHRMHN4MBSWIH7HZQ, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:45.910012327 (590h34m45.781095372s) [noise=true , address=145.79.227.5:4223 , piece_id=M52N3HJ37KOPA3LNGSIYM5V57YE4ELOONXUMD54JNIUJLMYGHZHQ, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:45.914200573 (590h34m45.785283618s) [noise=true , address=145.79.227.10:4402 , piece_id=RB4ELKAZWOZYJFPVJSSQSQ4A6MQ3QABD64ZOFBMBGBLP4ZBLYTCQ, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:45.930777117 (590h34m45.801860162s) [noise=true , address=145.79.227.8:4329 , piece_id=2U24RSR6RO4AUNPUKIL2EHPXFIIJV6J54Q5XV37Z7DUJ47YVLUOA, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:45.931789354 (590h34m45.802872399s) [noise=true , address=145.79.227.15:4873 , piece_id=G2V6NLM4Q2JGZOB7CIX224JYFQUXPCQXR6YR3YZ7HU7QNP7LPQRA, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:45.933085292 (590h34m45.804168337s) [noise=true , address=145.79.227.14:4837 , piece_id=2RR2YOJE4Z2HPO662QZMAH4BWUJIVP26T3LUHM4N6Y3CQEPWDB7Q, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:45.945017462 (590h34m45.816100497s) [noise=true , address=145.79.227.12:4765 , piece_id=IFHTZWQASQV57XXQ4CK4OBDCMCZLFNWA6MK7B3J5SOGDUGKLQ75A, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:45.997781494 (590h34m45.868864539s) [noise=true , address=145.79.227.11:4731 , piece_id=TIIT6OYWB5WSC2YW44KYDOVM4RLZXFAQDSDVKSRBCBDYJ5XRZRNA, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:46.006849841 (590h34m45.877932876s) [noise=true , address=145.79.227.38:4978 , piece_id=TCCP6WCVDGLV7DRNMZLZFO5FTGLIV2CWEUEOWJ7LW3GJMMASPZMQ, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:46.225565652 (590h34m46.096648687s) [noise=true , address=77.90.31.14:4664 , piece_id=V6BZMBCU5Z6ZKUXNZ7E7AGRTKUNTMDEA3532BVP5K2DJKRTDXK6Q, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:46.265443075 (590h34m46.13652612s) [noise=true , address=45.91.116.5:4297 , piece_id=BC76YOSI7JD6KIC6O2ZIH5EAEQSRUI5TQ33RKEC53PAQA7Q2OLIQ, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:46.600316007 (590h34m46.471399052s) [noise=true , address=45.91.116.7:4012 , piece_id=ECNNYY6OAOBKZ55GGIX5E5YGMQKXUA6H4OPBMR3M32FYBO3NBXQA, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:46.801841444 (590h34m46.672924519s) [noise=true , address=77.90.31.18:4805 , piece_id=JDKZBALSEX6P4YGMWJWAI6ISCWC7DU5RKWCIJGIHK4YCTQOKMFOA, seg_pos={0 0}, upload=517732]: Done uploading piece. err: <nil> Dec 30 06:03:46.801867023 (590h34m46.672950058s) [seg_pos={0 0}, upload=517732]: Segment reached optimal threshold of 13 pieces. Dec 30 06:03:46.802099957 (590h34m46.673183002s) [noise=true , address=45.91.116.10:4084 , piece_id=SLRPH763LLXQQN4W7NO3W4UOTGXICMTBZC345F7GXKO5IBNZTOYA, seg_pos={0 0}, upload=517732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 30 06:03:46.802110587 (590h34m46.673193622s) [noise=true , address=45.91.116.10:4084 , piece_id=SLRPH763LLXQQN4W7NO3W4UOTGXICMTBZC345F7GXKO5IBNZTOYA, seg_pos={0 0}, upload=517732]: Long tail cancellation due to optimal threshold Dec 30 06:03:46.802131887 (590h34m46.673214932s) [noise=true , address=45.91.116.12:4225 , piece_id=WYVRJN2ERQ47GJHFNIX27A3VAPZH7UL3BH4YGFXU5MBCCW4DUVYQ, seg_pos={0 0}, upload=517732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 30 06:03:46.802139983 (590h34m46.673223038s) [noise=true , address=45.91.116.12:4225 , piece_id=WYVRJN2ERQ47GJHFNIX27A3VAPZH7UL3BH4YGFXU5MBCCW4DUVYQ, seg_pos={0 0}, upload=517732]: Long tail cancellation due to optimal threshold Dec 30 06:03:46.802472696 (590h34m46.673555741s) [noise=true , address=45.91.116.14:4190 , piece_id=5Q4LQPWJYNJTZA6KYOIXOHBYEJ3KZBYTMFC6DAV637PVJBMMJGCA, seg_pos={0 0}, upload=517732]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 30 06:03:46.802483287 (590h34m46.673566322s) [noise=true , address=45.91.116.14:4190 , piece_id=5Q4LQPWJYNJTZA6KYOIXOHBYEJ3KZBYTMFC6DAV637PVJBMMJGCA, seg_pos={0 0}, upload=517732]: Long tail cancellation due to optimal threshold Dec 30 06:03:46.802509205 (590h34m46.67359224s) [seg_pos={0 0}, upload=517732]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 06:03:46.802602804 (590h34m46.673685859s) [upload=517732]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 06:03:46.802610218 (590h34m46.673693263s) [upload=517732]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 06:03:47.099426405 (590h34m46.97050946s) [upload=517732]: Done uploading segments... Dec 30 06:03:47.099449608 (590h34m46.970532653s) [upload=517732]: Upload finished. err: <nil> Duration: 30.003419032s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 3.717333804s trace: https://jaeger.dev.storj.io/trace/51fe686d4b8ba83d FAIL (2025-12-30 06:04:06 UTC) orbiter warning: context deadline exceeded Duration: 3.717363141s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X..X...X....X.X.......X..X..X....X.XX.X..X..X..X..XX.......] duration: 6.075970322s trace: https://jaeger.dev.storj.io/trace/51c87eb35646907f FAIL (2025-12-30 06:03:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA) context deadline exceeded ======LOGS====== c 30 05:57:48.567806875 (590h28m48.43888992s) [upload=517647]: Starting upload Dec 30 05:57:48.567912356 (590h28m48.438995391s) [upload=517647]: Done starting upload Dec 30 05:57:48.567984994 (590h28m48.439068029s) [upload=517647]: Uploading segments... Dec 30 05:57:48.568173282 (590h28m48.439256327s) [upload=517647]: Got next segment. Inline: false Dec 30 05:57:48.568223909 (590h28m48.439306944s) [upload=517647]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:57:48.568233337 (590h28m48.439316382s) [upload=517647]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:57:48.807489480 (590h28m48.678572515s) [seg_pos={0 0}, upload=517647]: Begin upload segment... Dec 30 05:57:48.807529977 (590h28m48.678613012s) [seg_pos={0 0}, upload=517647]: Done begin upload segment. Dec 30 05:57:48.807553841 (590h28m48.678636876s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.811525425 (590h28m48.68260846s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.811564219 (590h28m48.682647264s) [noise=true , address=145.79.227.16:4911 , piece_id=HJTB3YGOURTBTT577QMKJBBFLWFMPY6NGTXEDKQ4XFOAF2RI6C7A, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.812873170 (590h28m48.683956206s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.812910992 (590h28m48.683994037s) [noise=true , address=145.79.227.11:4729 , piece_id=WSENCZD6EVDNA5JZGINYTMXF27YXTPO5OEUPSLKJBLFKIJNWTFAA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.814140935 (590h28m48.68522398s) [noise=true , address=145.79.227.19:4077 , piece_id=KWNDJ2LU2DJG2TLSTU4WVQ7HV6FZPVBQEURJSYWQKBXJ7RQ5DU6Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.814622252 (590h28m48.685705287s) [noise=true , address=145.79.227.38:4977 , piece_id=OONKWQ5VMVSQE2WU4WD62S7MG2AOE3NZQLX7IP22J4OB2ESFTVBA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.815333826 (590h28m48.686416861s) [noise=true , address=145.79.227.7:4294 , piece_id=DKWQOKXHNYISIZ4ND4WWKZD5ECVRDSK55J2SZ7DZCSDLEW7WLRRA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.815712347 (590h28m48.686795382s) [noise=true , address=145.79.227.22:4510 , piece_id=KF4ZTUPQ35KOU3DZ5QGWWYJMEMSRQI7COZIXUMCRMG457YE3IY7Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.816537719 (590h28m48.687620784s) [noise=true , address=77.90.31.15:4697 , piece_id=YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.817015598 (590h28m48.688098633s) [noise=true , address=145.79.227.49:4339 , piece_id=OLAR2GVVB4LAAA62PGUCNALLPXDRPA6MJ7N52UKNT4SO4GFV34CA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.817787278 (590h28m48.688870313s) [noise=true , address=145.79.227.9:4367 , piece_id=YMVFQIHUZ72XLLTUWLDE7XK6JBR43EEIZ5Y766CLZBRHJRWX7WSA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.818169556 (590h28m48.689252601s) [noise=true , address=77.90.31.10:4520 , piece_id=L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.819469922 (590h28m48.690552957s) [noise=true , address=145.79.227.48:4303 , piece_id=LH5GUHTSPVJDYI7XBOYD5QAMRBISZXKG2LAKPKDXEERQC7DWQ7JA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.821074578 (590h28m48.692157613s) [noise=true , address=45.91.116.12:4227 , piece_id=7ILRCBWTBV3NNCM2D25QXIDZNPLPN2QUDZRZZPIZL5F3KJQ232MA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.821461765 (590h28m48.69254481s) [noise=true , address=77.90.31.17:4770 , piece_id=QFXOPMSKXJIRJSHKA7R3JBJLGTGPYXJ2NUTGFTWHRHM6XFUW3VHQ, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:50.434782068 (590h28m50.305865113s) [upload=517647]: Next returned nil segment Dec 30 05:57:50.434794843 (590h28m50.305877878s) [upload=517647]: Waiting for error group managing segments... Dec 30 05:57:50.816278389 (590h28m50.687361424s) [noise=true , address=145.79.227.9:4367 , piece_id=YMVFQIHUZ72XLLTUWLDE7XK6JBR43EEIZ5Y766CLZBRHJRWX7WSA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.816522073 (590h28m50.687605108s) [noise=true , address=145.79.227.48:4303 , piece_id=LH5GUHTSPVJDYI7XBOYD5QAMRBISZXKG2LAKPKDXEERQC7DWQ7JA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.819180385 (590h28m50.69026342s) [noise=true , address=145.79.227.11:4729 , piece_id=WSENCZD6EVDNA5JZGINYTMXF27YXTPO5OEUPSLKJBLFKIJNWTFAA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.819709913 (590h28m50.690792958s) [noise=true , address=145.79.227.16:4911 , piece_id=HJTB3YGOURTBTT577QMKJBBFLWFMPY6NGTXEDKQ4XFOAF2RI6C7A, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.820513734 (590h28m50.691596769s) [noise=true , address=145.79.227.19:4077 , piece_id=KWNDJ2LU2DJG2TLSTU4WVQ7HV6FZPVBQEURJSYWQKBXJ7RQ5DU6Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.821350417 (590h28m50.692433462s) [noise=true , address=145.79.227.7:4294 , piece_id=DKWQOKXHNYISIZ4ND4WWKZD5ECVRDSK55J2SZ7DZCSDLEW7WLRRA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.824387480 (590h28m50.695470515s) [noise=true , address=145.79.227.22:4510 , piece_id=KF4ZTUPQ35KOU3DZ5QGWWYJMEMSRQI7COZIXUMCRMG457YE3IY7Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.892679318 (590h28m50.763762353s) [noise=true , address=145.79.227.38:4977 , piece_id=OONKWQ5VMVSQE2WU4WD62S7MG2AOE3NZQLX7IP22J4OB2ESFTVBA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.999047852 (590h28m50.870130887s) [noise=true , address=77.90.31.10:4520 , piece_id=L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.091779157 (590h28m50.962862192s) [noise=true , address=77.90.31.17:4770 , piece_id=QFXOPMSKXJIRJSHKA7R3JBJLGTGPYXJ2NUTGFTWHRHM6XFUW3VHQ, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.132348578 (590h28m51.003431613s) [noise=true , address=145.79.227.49:4339 , piece_id=OLAR2GVVB4LAAA62PGUCNALLPXDRPA6MJ7N52UKNT4SO4GFV34CA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.164327811 (590h28m51.035410846s) [noise=true , address=77.90.31.15:4697 , piece_id=YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.223011665 (590h28m51.09409471s) [noise=true , address=45.91.116.12:4227 , piece_id=7ILRCBWTBV3NNCM2D25QXIDZNPLPN2QUDZRZZPIZL5F3KJQ232MA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.223037134 (590h28m51.094120179s) [seg_pos={0 0}, upload=517647]: Segment reached optimal threshold of 13 pieces. Dec 30 05:57:51.223255058 (590h28m51.094338093s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 30 05:57:51.223265369 (590h28m51.094348414s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223377812 (590h28m51.094460858s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 05:57:51.223390166 (590h28m51.094473211s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223420163 (590h28m51.094503208s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection Dec 30 05:57:51.223428138 (590h28m51.094511173s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223450210 (590h28m51.094533245s) [seg_pos={0 0}, upload=517647]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:57:51.223530653 (590h28m51.094613698s) [upload=517647]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:57:51.223536685 (590h28m51.094619741s) [upload=517647]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:57:51.515111610 (590h28m51.386194646s) [upload=517647]: Done uploading segments... Dec 30 05:57:51.515157619 (590h28m51.386240664s) [upload=517647]: Upload finished. err: <nil> Duration: 6.076009457s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 3.413091029s trace: https://jaeger.dev.storj.io/trace/51fe686d4b8ba83d FAIL (2025-12-30 06:03:13 UTC) orbiter warning: context deadline exceeded Duration: 3.413115534s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001168582s trace: https://jaeger.dev.storj.io/trace/51fe686d4b8ba83d FAIL (2025-12-30 06:02:56 UTC) orbiter warning: context deadline exceeded Duration: 5.001192788s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..X..X...X....X.X.......X..X..X....X.XX.X..X..X..X..XX......] duration: 5.945563877s trace: https://jaeger.dev.storj.io/trace/51c87eb35646907f FAIL (2025-12-30 06:02:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA) context deadline exceeded ======LOGS====== c 30 05:57:48.567806875 (590h28m48.43888992s) [upload=517647]: Starting upload Dec 30 05:57:48.567912356 (590h28m48.438995391s) [upload=517647]: Done starting upload Dec 30 05:57:48.567984994 (590h28m48.439068029s) [upload=517647]: Uploading segments... Dec 30 05:57:48.568173282 (590h28m48.439256327s) [upload=517647]: Got next segment. Inline: false Dec 30 05:57:48.568223909 (590h28m48.439306944s) [upload=517647]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:57:48.568233337 (590h28m48.439316382s) [upload=517647]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:57:48.807489480 (590h28m48.678572515s) [seg_pos={0 0}, upload=517647]: Begin upload segment... Dec 30 05:57:48.807529977 (590h28m48.678613012s) [seg_pos={0 0}, upload=517647]: Done begin upload segment. Dec 30 05:57:48.807553841 (590h28m48.678636876s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.811525425 (590h28m48.68260846s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.811564219 (590h28m48.682647264s) [noise=true , address=145.79.227.16:4911 , piece_id=HJTB3YGOURTBTT577QMKJBBFLWFMPY6NGTXEDKQ4XFOAF2RI6C7A, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.812873170 (590h28m48.683956206s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.812910992 (590h28m48.683994037s) [noise=true , address=145.79.227.11:4729 , piece_id=WSENCZD6EVDNA5JZGINYTMXF27YXTPO5OEUPSLKJBLFKIJNWTFAA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.814140935 (590h28m48.68522398s) [noise=true , address=145.79.227.19:4077 , piece_id=KWNDJ2LU2DJG2TLSTU4WVQ7HV6FZPVBQEURJSYWQKBXJ7RQ5DU6Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.814622252 (590h28m48.685705287s) [noise=true , address=145.79.227.38:4977 , piece_id=OONKWQ5VMVSQE2WU4WD62S7MG2AOE3NZQLX7IP22J4OB2ESFTVBA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.815333826 (590h28m48.686416861s) [noise=true , address=145.79.227.7:4294 , piece_id=DKWQOKXHNYISIZ4ND4WWKZD5ECVRDSK55J2SZ7DZCSDLEW7WLRRA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.815712347 (590h28m48.686795382s) [noise=true , address=145.79.227.22:4510 , piece_id=KF4ZTUPQ35KOU3DZ5QGWWYJMEMSRQI7COZIXUMCRMG457YE3IY7Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.816537719 (590h28m48.687620784s) [noise=true , address=77.90.31.15:4697 , piece_id=YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.817015598 (590h28m48.688098633s) [noise=true , address=145.79.227.49:4339 , piece_id=OLAR2GVVB4LAAA62PGUCNALLPXDRPA6MJ7N52UKNT4SO4GFV34CA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.817787278 (590h28m48.688870313s) [noise=true , address=145.79.227.9:4367 , piece_id=YMVFQIHUZ72XLLTUWLDE7XK6JBR43EEIZ5Y766CLZBRHJRWX7WSA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.818169556 (590h28m48.689252601s) [noise=true , address=77.90.31.10:4520 , piece_id=L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.819469922 (590h28m48.690552957s) [noise=true , address=145.79.227.48:4303 , piece_id=LH5GUHTSPVJDYI7XBOYD5QAMRBISZXKG2LAKPKDXEERQC7DWQ7JA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.821074578 (590h28m48.692157613s) [noise=true , address=45.91.116.12:4227 , piece_id=7ILRCBWTBV3NNCM2D25QXIDZNPLPN2QUDZRZZPIZL5F3KJQ232MA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.821461765 (590h28m48.69254481s) [noise=true , address=77.90.31.17:4770 , piece_id=QFXOPMSKXJIRJSHKA7R3JBJLGTGPYXJ2NUTGFTWHRHM6XFUW3VHQ, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:50.434782068 (590h28m50.305865113s) [upload=517647]: Next returned nil segment Dec 30 05:57:50.434794843 (590h28m50.305877878s) [upload=517647]: Waiting for error group managing segments... Dec 30 05:57:50.816278389 (590h28m50.687361424s) [noise=true , address=145.79.227.9:4367 , piece_id=YMVFQIHUZ72XLLTUWLDE7XK6JBR43EEIZ5Y766CLZBRHJRWX7WSA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.816522073 (590h28m50.687605108s) [noise=true , address=145.79.227.48:4303 , piece_id=LH5GUHTSPVJDYI7XBOYD5QAMRBISZXKG2LAKPKDXEERQC7DWQ7JA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.819180385 (590h28m50.69026342s) [noise=true , address=145.79.227.11:4729 , piece_id=WSENCZD6EVDNA5JZGINYTMXF27YXTPO5OEUPSLKJBLFKIJNWTFAA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.819709913 (590h28m50.690792958s) [noise=true , address=145.79.227.16:4911 , piece_id=HJTB3YGOURTBTT577QMKJBBFLWFMPY6NGTXEDKQ4XFOAF2RI6C7A, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.820513734 (590h28m50.691596769s) [noise=true , address=145.79.227.19:4077 , piece_id=KWNDJ2LU2DJG2TLSTU4WVQ7HV6FZPVBQEURJSYWQKBXJ7RQ5DU6Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.821350417 (590h28m50.692433462s) [noise=true , address=145.79.227.7:4294 , piece_id=DKWQOKXHNYISIZ4ND4WWKZD5ECVRDSK55J2SZ7DZCSDLEW7WLRRA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.824387480 (590h28m50.695470515s) [noise=true , address=145.79.227.22:4510 , piece_id=KF4ZTUPQ35KOU3DZ5QGWWYJMEMSRQI7COZIXUMCRMG457YE3IY7Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.892679318 (590h28m50.763762353s) [noise=true , address=145.79.227.38:4977 , piece_id=OONKWQ5VMVSQE2WU4WD62S7MG2AOE3NZQLX7IP22J4OB2ESFTVBA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.999047852 (590h28m50.870130887s) [noise=true , address=77.90.31.10:4520 , piece_id=L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.091779157 (590h28m50.962862192s) [noise=true , address=77.90.31.17:4770 , piece_id=QFXOPMSKXJIRJSHKA7R3JBJLGTGPYXJ2NUTGFTWHRHM6XFUW3VHQ, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.132348578 (590h28m51.003431613s) [noise=true , address=145.79.227.49:4339 , piece_id=OLAR2GVVB4LAAA62PGUCNALLPXDRPA6MJ7N52UKNT4SO4GFV34CA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.164327811 (590h28m51.035410846s) [noise=true , address=77.90.31.15:4697 , piece_id=YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.223011665 (590h28m51.09409471s) [noise=true , address=45.91.116.12:4227 , piece_id=7ILRCBWTBV3NNCM2D25QXIDZNPLPN2QUDZRZZPIZL5F3KJQ232MA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.223037134 (590h28m51.094120179s) [seg_pos={0 0}, upload=517647]: Segment reached optimal threshold of 13 pieces. Dec 30 05:57:51.223255058 (590h28m51.094338093s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 30 05:57:51.223265369 (590h28m51.094348414s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223377812 (590h28m51.094460858s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 05:57:51.223390166 (590h28m51.094473211s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223420163 (590h28m51.094503208s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection Dec 30 05:57:51.223428138 (590h28m51.094511173s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223450210 (590h28m51.094533245s) [seg_pos={0 0}, upload=517647]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:57:51.223530653 (590h28m51.094613698s) [upload=517647]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:57:51.223536685 (590h28m51.094619741s) [upload=517647]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:57:51.515111610 (590h28m51.386194646s) [upload=517647]: Done uploading segments... Dec 30 05:57:51.515157619 (590h28m51.386240664s) [upload=517647]: Upload finished. err: <nil> Duration: 5.945588264s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 4.66313794s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 06:01:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.663168547s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X..X..X...X....X.X.......X..X..X....X.XX.X..X..X..X..XX.....] duration: 12.221682745s trace: https://jaeger.dev.storj.io/trace/51c87eb35646907f FAIL (2025-12-30 06:01:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA) context deadline exceeded ======LOGS====== c 30 05:57:48.567806875 (590h28m48.43888992s) [upload=517647]: Starting upload Dec 30 05:57:48.567912356 (590h28m48.438995391s) [upload=517647]: Done starting upload Dec 30 05:57:48.567984994 (590h28m48.439068029s) [upload=517647]: Uploading segments... Dec 30 05:57:48.568173282 (590h28m48.439256327s) [upload=517647]: Got next segment. Inline: false Dec 30 05:57:48.568223909 (590h28m48.439306944s) [upload=517647]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:57:48.568233337 (590h28m48.439316382s) [upload=517647]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:57:48.807489480 (590h28m48.678572515s) [seg_pos={0 0}, upload=517647]: Begin upload segment... Dec 30 05:57:48.807529977 (590h28m48.678613012s) [seg_pos={0 0}, upload=517647]: Done begin upload segment. Dec 30 05:57:48.807553841 (590h28m48.678636876s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.811525425 (590h28m48.68260846s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.811564219 (590h28m48.682647264s) [noise=true , address=145.79.227.16:4911 , piece_id=HJTB3YGOURTBTT577QMKJBBFLWFMPY6NGTXEDKQ4XFOAF2RI6C7A, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.812873170 (590h28m48.683956206s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.812910992 (590h28m48.683994037s) [noise=true , address=145.79.227.11:4729 , piece_id=WSENCZD6EVDNA5JZGINYTMXF27YXTPO5OEUPSLKJBLFKIJNWTFAA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.814140935 (590h28m48.68522398s) [noise=true , address=145.79.227.19:4077 , piece_id=KWNDJ2LU2DJG2TLSTU4WVQ7HV6FZPVBQEURJSYWQKBXJ7RQ5DU6Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.814622252 (590h28m48.685705287s) [noise=true , address=145.79.227.38:4977 , piece_id=OONKWQ5VMVSQE2WU4WD62S7MG2AOE3NZQLX7IP22J4OB2ESFTVBA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.815333826 (590h28m48.686416861s) [noise=true , address=145.79.227.7:4294 , piece_id=DKWQOKXHNYISIZ4ND4WWKZD5ECVRDSK55J2SZ7DZCSDLEW7WLRRA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.815712347 (590h28m48.686795382s) [noise=true , address=145.79.227.22:4510 , piece_id=KF4ZTUPQ35KOU3DZ5QGWWYJMEMSRQI7COZIXUMCRMG457YE3IY7Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.816537719 (590h28m48.687620784s) [noise=true , address=77.90.31.15:4697 , piece_id=YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.817015598 (590h28m48.688098633s) [noise=true , address=145.79.227.49:4339 , piece_id=OLAR2GVVB4LAAA62PGUCNALLPXDRPA6MJ7N52UKNT4SO4GFV34CA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.817787278 (590h28m48.688870313s) [noise=true , address=145.79.227.9:4367 , piece_id=YMVFQIHUZ72XLLTUWLDE7XK6JBR43EEIZ5Y766CLZBRHJRWX7WSA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.818169556 (590h28m48.689252601s) [noise=true , address=77.90.31.10:4520 , piece_id=L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.819469922 (590h28m48.690552957s) [noise=true , address=145.79.227.48:4303 , piece_id=LH5GUHTSPVJDYI7XBOYD5QAMRBISZXKG2LAKPKDXEERQC7DWQ7JA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.821074578 (590h28m48.692157613s) [noise=true , address=45.91.116.12:4227 , piece_id=7ILRCBWTBV3NNCM2D25QXIDZNPLPN2QUDZRZZPIZL5F3KJQ232MA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.821461765 (590h28m48.69254481s) [noise=true , address=77.90.31.17:4770 , piece_id=QFXOPMSKXJIRJSHKA7R3JBJLGTGPYXJ2NUTGFTWHRHM6XFUW3VHQ, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:50.434782068 (590h28m50.305865113s) [upload=517647]: Next returned nil segment Dec 30 05:57:50.434794843 (590h28m50.305877878s) [upload=517647]: Waiting for error group managing segments... Dec 30 05:57:50.816278389 (590h28m50.687361424s) [noise=true , address=145.79.227.9:4367 , piece_id=YMVFQIHUZ72XLLTUWLDE7XK6JBR43EEIZ5Y766CLZBRHJRWX7WSA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.816522073 (590h28m50.687605108s) [noise=true , address=145.79.227.48:4303 , piece_id=LH5GUHTSPVJDYI7XBOYD5QAMRBISZXKG2LAKPKDXEERQC7DWQ7JA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.819180385 (590h28m50.69026342s) [noise=true , address=145.79.227.11:4729 , piece_id=WSENCZD6EVDNA5JZGINYTMXF27YXTPO5OEUPSLKJBLFKIJNWTFAA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.819709913 (590h28m50.690792958s) [noise=true , address=145.79.227.16:4911 , piece_id=HJTB3YGOURTBTT577QMKJBBFLWFMPY6NGTXEDKQ4XFOAF2RI6C7A, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.820513734 (590h28m50.691596769s) [noise=true , address=145.79.227.19:4077 , piece_id=KWNDJ2LU2DJG2TLSTU4WVQ7HV6FZPVBQEURJSYWQKBXJ7RQ5DU6Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.821350417 (590h28m50.692433462s) [noise=true , address=145.79.227.7:4294 , piece_id=DKWQOKXHNYISIZ4ND4WWKZD5ECVRDSK55J2SZ7DZCSDLEW7WLRRA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.824387480 (590h28m50.695470515s) [noise=true , address=145.79.227.22:4510 , piece_id=KF4ZTUPQ35KOU3DZ5QGWWYJMEMSRQI7COZIXUMCRMG457YE3IY7Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.892679318 (590h28m50.763762353s) [noise=true , address=145.79.227.38:4977 , piece_id=OONKWQ5VMVSQE2WU4WD62S7MG2AOE3NZQLX7IP22J4OB2ESFTVBA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.999047852 (590h28m50.870130887s) [noise=true , address=77.90.31.10:4520 , piece_id=L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.091779157 (590h28m50.962862192s) [noise=true , address=77.90.31.17:4770 , piece_id=QFXOPMSKXJIRJSHKA7R3JBJLGTGPYXJ2NUTGFTWHRHM6XFUW3VHQ, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.132348578 (590h28m51.003431613s) [noise=true , address=145.79.227.49:4339 , piece_id=OLAR2GVVB4LAAA62PGUCNALLPXDRPA6MJ7N52UKNT4SO4GFV34CA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.164327811 (590h28m51.035410846s) [noise=true , address=77.90.31.15:4697 , piece_id=YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.223011665 (590h28m51.09409471s) [noise=true , address=45.91.116.12:4227 , piece_id=7ILRCBWTBV3NNCM2D25QXIDZNPLPN2QUDZRZZPIZL5F3KJQ232MA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.223037134 (590h28m51.094120179s) [seg_pos={0 0}, upload=517647]: Segment reached optimal threshold of 13 pieces. Dec 30 05:57:51.223255058 (590h28m51.094338093s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 30 05:57:51.223265369 (590h28m51.094348414s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223377812 (590h28m51.094460858s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 05:57:51.223390166 (590h28m51.094473211s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223420163 (590h28m51.094503208s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection Dec 30 05:57:51.223428138 (590h28m51.094511173s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223450210 (590h28m51.094533245s) [seg_pos={0 0}, upload=517647]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:57:51.223530653 (590h28m51.094613698s) [upload=517647]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:57:51.223536685 (590h28m51.094619741s) [upload=517647]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:57:51.515111610 (590h28m51.386194646s) [upload=517647]: Done uploading segments... Dec 30 05:57:51.515157619 (590h28m51.386240664s) [upload=517647]: Upload finished. err: <nil> Duration: 12.221708213s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XX..X..X...X....X.X.......X..X..X....X.XX.X..X..X..X..XX....] duration: 5.797011388s trace: https://jaeger.dev.storj.io/trace/51c87eb35646907f FAIL (2025-12-30 06:01:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA) context deadline exceeded ======LOGS====== c 30 05:57:48.567806875 (590h28m48.43888992s) [upload=517647]: Starting upload Dec 30 05:57:48.567912356 (590h28m48.438995391s) [upload=517647]: Done starting upload Dec 30 05:57:48.567984994 (590h28m48.439068029s) [upload=517647]: Uploading segments... Dec 30 05:57:48.568173282 (590h28m48.439256327s) [upload=517647]: Got next segment. Inline: false Dec 30 05:57:48.568223909 (590h28m48.439306944s) [upload=517647]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:57:48.568233337 (590h28m48.439316382s) [upload=517647]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:57:48.807489480 (590h28m48.678572515s) [seg_pos={0 0}, upload=517647]: Begin upload segment... Dec 30 05:57:48.807529977 (590h28m48.678613012s) [seg_pos={0 0}, upload=517647]: Done begin upload segment. Dec 30 05:57:48.807553841 (590h28m48.678636876s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.811525425 (590h28m48.68260846s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.811564219 (590h28m48.682647264s) [noise=true , address=145.79.227.16:4911 , piece_id=HJTB3YGOURTBTT577QMKJBBFLWFMPY6NGTXEDKQ4XFOAF2RI6C7A, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.812873170 (590h28m48.683956206s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.812910992 (590h28m48.683994037s) [noise=true , address=145.79.227.11:4729 , piece_id=WSENCZD6EVDNA5JZGINYTMXF27YXTPO5OEUPSLKJBLFKIJNWTFAA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.814140935 (590h28m48.68522398s) [noise=true , address=145.79.227.19:4077 , piece_id=KWNDJ2LU2DJG2TLSTU4WVQ7HV6FZPVBQEURJSYWQKBXJ7RQ5DU6Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.814622252 (590h28m48.685705287s) [noise=true , address=145.79.227.38:4977 , piece_id=OONKWQ5VMVSQE2WU4WD62S7MG2AOE3NZQLX7IP22J4OB2ESFTVBA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.815333826 (590h28m48.686416861s) [noise=true , address=145.79.227.7:4294 , piece_id=DKWQOKXHNYISIZ4ND4WWKZD5ECVRDSK55J2SZ7DZCSDLEW7WLRRA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.815712347 (590h28m48.686795382s) [noise=true , address=145.79.227.22:4510 , piece_id=KF4ZTUPQ35KOU3DZ5QGWWYJMEMSRQI7COZIXUMCRMG457YE3IY7Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.816537719 (590h28m48.687620784s) [noise=true , address=77.90.31.15:4697 , piece_id=YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.817015598 (590h28m48.688098633s) [noise=true , address=145.79.227.49:4339 , piece_id=OLAR2GVVB4LAAA62PGUCNALLPXDRPA6MJ7N52UKNT4SO4GFV34CA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.817787278 (590h28m48.688870313s) [noise=true , address=145.79.227.9:4367 , piece_id=YMVFQIHUZ72XLLTUWLDE7XK6JBR43EEIZ5Y766CLZBRHJRWX7WSA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.818169556 (590h28m48.689252601s) [noise=true , address=77.90.31.10:4520 , piece_id=L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.819469922 (590h28m48.690552957s) [noise=true , address=145.79.227.48:4303 , piece_id=LH5GUHTSPVJDYI7XBOYD5QAMRBISZXKG2LAKPKDXEERQC7DWQ7JA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.821074578 (590h28m48.692157613s) [noise=true , address=45.91.116.12:4227 , piece_id=7ILRCBWTBV3NNCM2D25QXIDZNPLPN2QUDZRZZPIZL5F3KJQ232MA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.821461765 (590h28m48.69254481s) [noise=true , address=77.90.31.17:4770 , piece_id=QFXOPMSKXJIRJSHKA7R3JBJLGTGPYXJ2NUTGFTWHRHM6XFUW3VHQ, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:50.434782068 (590h28m50.305865113s) [upload=517647]: Next returned nil segment Dec 30 05:57:50.434794843 (590h28m50.305877878s) [upload=517647]: Waiting for error group managing segments... Dec 30 05:57:50.816278389 (590h28m50.687361424s) [noise=true , address=145.79.227.9:4367 , piece_id=YMVFQIHUZ72XLLTUWLDE7XK6JBR43EEIZ5Y766CLZBRHJRWX7WSA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.816522073 (590h28m50.687605108s) [noise=true , address=145.79.227.48:4303 , piece_id=LH5GUHTSPVJDYI7XBOYD5QAMRBISZXKG2LAKPKDXEERQC7DWQ7JA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.819180385 (590h28m50.69026342s) [noise=true , address=145.79.227.11:4729 , piece_id=WSENCZD6EVDNA5JZGINYTMXF27YXTPO5OEUPSLKJBLFKIJNWTFAA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.819709913 (590h28m50.690792958s) [noise=true , address=145.79.227.16:4911 , piece_id=HJTB3YGOURTBTT577QMKJBBFLWFMPY6NGTXEDKQ4XFOAF2RI6C7A, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.820513734 (590h28m50.691596769s) [noise=true , address=145.79.227.19:4077 , piece_id=KWNDJ2LU2DJG2TLSTU4WVQ7HV6FZPVBQEURJSYWQKBXJ7RQ5DU6Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.821350417 (590h28m50.692433462s) [noise=true , address=145.79.227.7:4294 , piece_id=DKWQOKXHNYISIZ4ND4WWKZD5ECVRDSK55J2SZ7DZCSDLEW7WLRRA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.824387480 (590h28m50.695470515s) [noise=true , address=145.79.227.22:4510 , piece_id=KF4ZTUPQ35KOU3DZ5QGWWYJMEMSRQI7COZIXUMCRMG457YE3IY7Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.892679318 (590h28m50.763762353s) [noise=true , address=145.79.227.38:4977 , piece_id=OONKWQ5VMVSQE2WU4WD62S7MG2AOE3NZQLX7IP22J4OB2ESFTVBA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.999047852 (590h28m50.870130887s) [noise=true , address=77.90.31.10:4520 , piece_id=L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.091779157 (590h28m50.962862192s) [noise=true , address=77.90.31.17:4770 , piece_id=QFXOPMSKXJIRJSHKA7R3JBJLGTGPYXJ2NUTGFTWHRHM6XFUW3VHQ, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.132348578 (590h28m51.003431613s) [noise=true , address=145.79.227.49:4339 , piece_id=OLAR2GVVB4LAAA62PGUCNALLPXDRPA6MJ7N52UKNT4SO4GFV34CA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.164327811 (590h28m51.035410846s) [noise=true , address=77.90.31.15:4697 , piece_id=YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.223011665 (590h28m51.09409471s) [noise=true , address=45.91.116.12:4227 , piece_id=7ILRCBWTBV3NNCM2D25QXIDZNPLPN2QUDZRZZPIZL5F3KJQ232MA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.223037134 (590h28m51.094120179s) [seg_pos={0 0}, upload=517647]: Segment reached optimal threshold of 13 pieces. Dec 30 05:57:51.223255058 (590h28m51.094338093s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 30 05:57:51.223265369 (590h28m51.094348414s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223377812 (590h28m51.094460858s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 05:57:51.223390166 (590h28m51.094473211s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223420163 (590h28m51.094503208s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection Dec 30 05:57:51.223428138 (590h28m51.094511173s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223450210 (590h28m51.094533245s) [seg_pos={0 0}, upload=517647]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:57:51.223530653 (590h28m51.094613698s) [upload=517647]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:57:51.223536685 (590h28m51.094619741s) [upload=517647]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:57:51.515111610 (590h28m51.386194646s) [upload=517647]: Done uploading segments... Dec 30 05:57:51.515157619 (590h28m51.386240664s) [upload=517647]: Upload finished. err: <nil> Duration: 5.797045032s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.423401851s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 06:00:52 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.423582826s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XXX..X..X...X....X.X.......X..X..X....X.XX.X..X..X..X..XX...] duration: 6.138223834s trace: https://jaeger.dev.storj.io/trace/51c87eb35646907f FAIL (2025-12-30 06:00:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA) context deadline exceeded ======LOGS====== c 30 05:57:48.567806875 (590h28m48.43888992s) [upload=517647]: Starting upload Dec 30 05:57:48.567912356 (590h28m48.438995391s) [upload=517647]: Done starting upload Dec 30 05:57:48.567984994 (590h28m48.439068029s) [upload=517647]: Uploading segments... Dec 30 05:57:48.568173282 (590h28m48.439256327s) [upload=517647]: Got next segment. Inline: false Dec 30 05:57:48.568223909 (590h28m48.439306944s) [upload=517647]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:57:48.568233337 (590h28m48.439316382s) [upload=517647]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:57:48.807489480 (590h28m48.678572515s) [seg_pos={0 0}, upload=517647]: Begin upload segment... Dec 30 05:57:48.807529977 (590h28m48.678613012s) [seg_pos={0 0}, upload=517647]: Done begin upload segment. Dec 30 05:57:48.807553841 (590h28m48.678636876s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.811525425 (590h28m48.68260846s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.811564219 (590h28m48.682647264s) [noise=true , address=145.79.227.16:4911 , piece_id=HJTB3YGOURTBTT577QMKJBBFLWFMPY6NGTXEDKQ4XFOAF2RI6C7A, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.812873170 (590h28m48.683956206s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.812910992 (590h28m48.683994037s) [noise=true , address=145.79.227.11:4729 , piece_id=WSENCZD6EVDNA5JZGINYTMXF27YXTPO5OEUPSLKJBLFKIJNWTFAA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.814140935 (590h28m48.68522398s) [noise=true , address=145.79.227.19:4077 , piece_id=KWNDJ2LU2DJG2TLSTU4WVQ7HV6FZPVBQEURJSYWQKBXJ7RQ5DU6Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.814622252 (590h28m48.685705287s) [noise=true , address=145.79.227.38:4977 , piece_id=OONKWQ5VMVSQE2WU4WD62S7MG2AOE3NZQLX7IP22J4OB2ESFTVBA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.815333826 (590h28m48.686416861s) [noise=true , address=145.79.227.7:4294 , piece_id=DKWQOKXHNYISIZ4ND4WWKZD5ECVRDSK55J2SZ7DZCSDLEW7WLRRA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.815712347 (590h28m48.686795382s) [noise=true , address=145.79.227.22:4510 , piece_id=KF4ZTUPQ35KOU3DZ5QGWWYJMEMSRQI7COZIXUMCRMG457YE3IY7Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.816537719 (590h28m48.687620784s) [noise=true , address=77.90.31.15:4697 , piece_id=YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.817015598 (590h28m48.688098633s) [noise=true , address=145.79.227.49:4339 , piece_id=OLAR2GVVB4LAAA62PGUCNALLPXDRPA6MJ7N52UKNT4SO4GFV34CA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.817787278 (590h28m48.688870313s) [noise=true , address=145.79.227.9:4367 , piece_id=YMVFQIHUZ72XLLTUWLDE7XK6JBR43EEIZ5Y766CLZBRHJRWX7WSA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.818169556 (590h28m48.689252601s) [noise=true , address=77.90.31.10:4520 , piece_id=L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.819469922 (590h28m48.690552957s) [noise=true , address=145.79.227.48:4303 , piece_id=LH5GUHTSPVJDYI7XBOYD5QAMRBISZXKG2LAKPKDXEERQC7DWQ7JA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.821074578 (590h28m48.692157613s) [noise=true , address=45.91.116.12:4227 , piece_id=7ILRCBWTBV3NNCM2D25QXIDZNPLPN2QUDZRZZPIZL5F3KJQ232MA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.821461765 (590h28m48.69254481s) [noise=true , address=77.90.31.17:4770 , piece_id=QFXOPMSKXJIRJSHKA7R3JBJLGTGPYXJ2NUTGFTWHRHM6XFUW3VHQ, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:50.434782068 (590h28m50.305865113s) [upload=517647]: Next returned nil segment Dec 30 05:57:50.434794843 (590h28m50.305877878s) [upload=517647]: Waiting for error group managing segments... Dec 30 05:57:50.816278389 (590h28m50.687361424s) [noise=true , address=145.79.227.9:4367 , piece_id=YMVFQIHUZ72XLLTUWLDE7XK6JBR43EEIZ5Y766CLZBRHJRWX7WSA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.816522073 (590h28m50.687605108s) [noise=true , address=145.79.227.48:4303 , piece_id=LH5GUHTSPVJDYI7XBOYD5QAMRBISZXKG2LAKPKDXEERQC7DWQ7JA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.819180385 (590h28m50.69026342s) [noise=true , address=145.79.227.11:4729 , piece_id=WSENCZD6EVDNA5JZGINYTMXF27YXTPO5OEUPSLKJBLFKIJNWTFAA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.819709913 (590h28m50.690792958s) [noise=true , address=145.79.227.16:4911 , piece_id=HJTB3YGOURTBTT577QMKJBBFLWFMPY6NGTXEDKQ4XFOAF2RI6C7A, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.820513734 (590h28m50.691596769s) [noise=true , address=145.79.227.19:4077 , piece_id=KWNDJ2LU2DJG2TLSTU4WVQ7HV6FZPVBQEURJSYWQKBXJ7RQ5DU6Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.821350417 (590h28m50.692433462s) [noise=true , address=145.79.227.7:4294 , piece_id=DKWQOKXHNYISIZ4ND4WWKZD5ECVRDSK55J2SZ7DZCSDLEW7WLRRA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.824387480 (590h28m50.695470515s) [noise=true , address=145.79.227.22:4510 , piece_id=KF4ZTUPQ35KOU3DZ5QGWWYJMEMSRQI7COZIXUMCRMG457YE3IY7Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.892679318 (590h28m50.763762353s) [noise=true , address=145.79.227.38:4977 , piece_id=OONKWQ5VMVSQE2WU4WD62S7MG2AOE3NZQLX7IP22J4OB2ESFTVBA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.999047852 (590h28m50.870130887s) [noise=true , address=77.90.31.10:4520 , piece_id=L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.091779157 (590h28m50.962862192s) [noise=true , address=77.90.31.17:4770 , piece_id=QFXOPMSKXJIRJSHKA7R3JBJLGTGPYXJ2NUTGFTWHRHM6XFUW3VHQ, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.132348578 (590h28m51.003431613s) [noise=true , address=145.79.227.49:4339 , piece_id=OLAR2GVVB4LAAA62PGUCNALLPXDRPA6MJ7N52UKNT4SO4GFV34CA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.164327811 (590h28m51.035410846s) [noise=true , address=77.90.31.15:4697 , piece_id=YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.223011665 (590h28m51.09409471s) [noise=true , address=45.91.116.12:4227 , piece_id=7ILRCBWTBV3NNCM2D25QXIDZNPLPN2QUDZRZZPIZL5F3KJQ232MA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.223037134 (590h28m51.094120179s) [seg_pos={0 0}, upload=517647]: Segment reached optimal threshold of 13 pieces. Dec 30 05:57:51.223255058 (590h28m51.094338093s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 30 05:57:51.223265369 (590h28m51.094348414s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223377812 (590h28m51.094460858s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 05:57:51.223390166 (590h28m51.094473211s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223420163 (590h28m51.094503208s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection Dec 30 05:57:51.223428138 (590h28m51.094511173s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223450210 (590h28m51.094533245s) [seg_pos={0 0}, upload=517647]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:57:51.223530653 (590h28m51.094613698s) [upload=517647]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:57:51.223536685 (590h28m51.094619741s) [upload=517647]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:57:51.515111610 (590h28m51.386194646s) [upload=517647]: Done uploading segments... Dec 30 05:57:51.515157619 (590h28m51.386240664s) [upload=517647]: Upload finished. err: <nil> Duration: 6.138283118s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.650910745s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 06:00:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.650932436s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXXX..X..X...X....X.X.......X..X..X....X.XX.X..X..X..X..XX..] duration: 6.020037076s trace: https://jaeger.dev.storj.io/trace/51c87eb35646907f FAIL (2025-12-30 05:59:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA) context deadline exceeded ======LOGS====== c 30 05:57:48.567806875 (590h28m48.43888992s) [upload=517647]: Starting upload Dec 30 05:57:48.567912356 (590h28m48.438995391s) [upload=517647]: Done starting upload Dec 30 05:57:48.567984994 (590h28m48.439068029s) [upload=517647]: Uploading segments... Dec 30 05:57:48.568173282 (590h28m48.439256327s) [upload=517647]: Got next segment. Inline: false Dec 30 05:57:48.568223909 (590h28m48.439306944s) [upload=517647]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:57:48.568233337 (590h28m48.439316382s) [upload=517647]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:57:48.807489480 (590h28m48.678572515s) [seg_pos={0 0}, upload=517647]: Begin upload segment... Dec 30 05:57:48.807529977 (590h28m48.678613012s) [seg_pos={0 0}, upload=517647]: Done begin upload segment. Dec 30 05:57:48.807553841 (590h28m48.678636876s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.811525425 (590h28m48.68260846s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.811564219 (590h28m48.682647264s) [noise=true , address=145.79.227.16:4911 , piece_id=HJTB3YGOURTBTT577QMKJBBFLWFMPY6NGTXEDKQ4XFOAF2RI6C7A, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.812873170 (590h28m48.683956206s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.812910992 (590h28m48.683994037s) [noise=true , address=145.79.227.11:4729 , piece_id=WSENCZD6EVDNA5JZGINYTMXF27YXTPO5OEUPSLKJBLFKIJNWTFAA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.814140935 (590h28m48.68522398s) [noise=true , address=145.79.227.19:4077 , piece_id=KWNDJ2LU2DJG2TLSTU4WVQ7HV6FZPVBQEURJSYWQKBXJ7RQ5DU6Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.814622252 (590h28m48.685705287s) [noise=true , address=145.79.227.38:4977 , piece_id=OONKWQ5VMVSQE2WU4WD62S7MG2AOE3NZQLX7IP22J4OB2ESFTVBA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.815333826 (590h28m48.686416861s) [noise=true , address=145.79.227.7:4294 , piece_id=DKWQOKXHNYISIZ4ND4WWKZD5ECVRDSK55J2SZ7DZCSDLEW7WLRRA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.815712347 (590h28m48.686795382s) [noise=true , address=145.79.227.22:4510 , piece_id=KF4ZTUPQ35KOU3DZ5QGWWYJMEMSRQI7COZIXUMCRMG457YE3IY7Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.816537719 (590h28m48.687620784s) [noise=true , address=77.90.31.15:4697 , piece_id=YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.817015598 (590h28m48.688098633s) [noise=true , address=145.79.227.49:4339 , piece_id=OLAR2GVVB4LAAA62PGUCNALLPXDRPA6MJ7N52UKNT4SO4GFV34CA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.817787278 (590h28m48.688870313s) [noise=true , address=145.79.227.9:4367 , piece_id=YMVFQIHUZ72XLLTUWLDE7XK6JBR43EEIZ5Y766CLZBRHJRWX7WSA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.818169556 (590h28m48.689252601s) [noise=true , address=77.90.31.10:4520 , piece_id=L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.819469922 (590h28m48.690552957s) [noise=true , address=145.79.227.48:4303 , piece_id=LH5GUHTSPVJDYI7XBOYD5QAMRBISZXKG2LAKPKDXEERQC7DWQ7JA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.821074578 (590h28m48.692157613s) [noise=true , address=45.91.116.12:4227 , piece_id=7ILRCBWTBV3NNCM2D25QXIDZNPLPN2QUDZRZZPIZL5F3KJQ232MA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.821461765 (590h28m48.69254481s) [noise=true , address=77.90.31.17:4770 , piece_id=QFXOPMSKXJIRJSHKA7R3JBJLGTGPYXJ2NUTGFTWHRHM6XFUW3VHQ, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:50.434782068 (590h28m50.305865113s) [upload=517647]: Next returned nil segment Dec 30 05:57:50.434794843 (590h28m50.305877878s) [upload=517647]: Waiting for error group managing segments... Dec 30 05:57:50.816278389 (590h28m50.687361424s) [noise=true , address=145.79.227.9:4367 , piece_id=YMVFQIHUZ72XLLTUWLDE7XK6JBR43EEIZ5Y766CLZBRHJRWX7WSA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.816522073 (590h28m50.687605108s) [noise=true , address=145.79.227.48:4303 , piece_id=LH5GUHTSPVJDYI7XBOYD5QAMRBISZXKG2LAKPKDXEERQC7DWQ7JA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.819180385 (590h28m50.69026342s) [noise=true , address=145.79.227.11:4729 , piece_id=WSENCZD6EVDNA5JZGINYTMXF27YXTPO5OEUPSLKJBLFKIJNWTFAA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.819709913 (590h28m50.690792958s) [noise=true , address=145.79.227.16:4911 , piece_id=HJTB3YGOURTBTT577QMKJBBFLWFMPY6NGTXEDKQ4XFOAF2RI6C7A, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.820513734 (590h28m50.691596769s) [noise=true , address=145.79.227.19:4077 , piece_id=KWNDJ2LU2DJG2TLSTU4WVQ7HV6FZPVBQEURJSYWQKBXJ7RQ5DU6Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.821350417 (590h28m50.692433462s) [noise=true , address=145.79.227.7:4294 , piece_id=DKWQOKXHNYISIZ4ND4WWKZD5ECVRDSK55J2SZ7DZCSDLEW7WLRRA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.824387480 (590h28m50.695470515s) [noise=true , address=145.79.227.22:4510 , piece_id=KF4ZTUPQ35KOU3DZ5QGWWYJMEMSRQI7COZIXUMCRMG457YE3IY7Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.892679318 (590h28m50.763762353s) [noise=true , address=145.79.227.38:4977 , piece_id=OONKWQ5VMVSQE2WU4WD62S7MG2AOE3NZQLX7IP22J4OB2ESFTVBA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.999047852 (590h28m50.870130887s) [noise=true , address=77.90.31.10:4520 , piece_id=L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.091779157 (590h28m50.962862192s) [noise=true , address=77.90.31.17:4770 , piece_id=QFXOPMSKXJIRJSHKA7R3JBJLGTGPYXJ2NUTGFTWHRHM6XFUW3VHQ, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.132348578 (590h28m51.003431613s) [noise=true , address=145.79.227.49:4339 , piece_id=OLAR2GVVB4LAAA62PGUCNALLPXDRPA6MJ7N52UKNT4SO4GFV34CA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.164327811 (590h28m51.035410846s) [noise=true , address=77.90.31.15:4697 , piece_id=YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.223011665 (590h28m51.09409471s) [noise=true , address=45.91.116.12:4227 , piece_id=7ILRCBWTBV3NNCM2D25QXIDZNPLPN2QUDZRZZPIZL5F3KJQ232MA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.223037134 (590h28m51.094120179s) [seg_pos={0 0}, upload=517647]: Segment reached optimal threshold of 13 pieces. Dec 30 05:57:51.223255058 (590h28m51.094338093s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 30 05:57:51.223265369 (590h28m51.094348414s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223377812 (590h28m51.094460858s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 05:57:51.223390166 (590h28m51.094473211s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223420163 (590h28m51.094503208s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection Dec 30 05:57:51.223428138 (590h28m51.094511173s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223450210 (590h28m51.094533245s) [seg_pos={0 0}, upload=517647]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:57:51.223530653 (590h28m51.094613698s) [upload=517647]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:57:51.223536685 (590h28m51.094619741s) [upload=517647]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:57:51.515111610 (590h28m51.386194646s) [upload=517647]: Done uploading segments... Dec 30 05:57:51.515157619 (590h28m51.386240664s) [upload=517647]: Upload finished. err: <nil> Duration: 6.020064268s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 4.895656702s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:59:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.895685188s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.430983828s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:59:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.43102173s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XXXXX..X..X...X....X.X.......X..X..X....X.XX.X..X..X..X..XX.] duration: 5.817613058s trace: https://jaeger.dev.storj.io/trace/51c87eb35646907f FAIL (2025-12-30 05:58:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA) context deadline exceeded ======LOGS====== c 30 05:57:48.567806875 (590h28m48.43888992s) [upload=517647]: Starting upload Dec 30 05:57:48.567912356 (590h28m48.438995391s) [upload=517647]: Done starting upload Dec 30 05:57:48.567984994 (590h28m48.439068029s) [upload=517647]: Uploading segments... Dec 30 05:57:48.568173282 (590h28m48.439256327s) [upload=517647]: Got next segment. Inline: false Dec 30 05:57:48.568223909 (590h28m48.439306944s) [upload=517647]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:57:48.568233337 (590h28m48.439316382s) [upload=517647]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:57:48.807489480 (590h28m48.678572515s) [seg_pos={0 0}, upload=517647]: Begin upload segment... Dec 30 05:57:48.807529977 (590h28m48.678613012s) [seg_pos={0 0}, upload=517647]: Done begin upload segment. Dec 30 05:57:48.807553841 (590h28m48.678636876s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.811525425 (590h28m48.68260846s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.811564219 (590h28m48.682647264s) [noise=true , address=145.79.227.16:4911 , piece_id=HJTB3YGOURTBTT577QMKJBBFLWFMPY6NGTXEDKQ4XFOAF2RI6C7A, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.812873170 (590h28m48.683956206s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.812910992 (590h28m48.683994037s) [noise=true , address=145.79.227.11:4729 , piece_id=WSENCZD6EVDNA5JZGINYTMXF27YXTPO5OEUPSLKJBLFKIJNWTFAA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.814140935 (590h28m48.68522398s) [noise=true , address=145.79.227.19:4077 , piece_id=KWNDJ2LU2DJG2TLSTU4WVQ7HV6FZPVBQEURJSYWQKBXJ7RQ5DU6Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.814622252 (590h28m48.685705287s) [noise=true , address=145.79.227.38:4977 , piece_id=OONKWQ5VMVSQE2WU4WD62S7MG2AOE3NZQLX7IP22J4OB2ESFTVBA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.815333826 (590h28m48.686416861s) [noise=true , address=145.79.227.7:4294 , piece_id=DKWQOKXHNYISIZ4ND4WWKZD5ECVRDSK55J2SZ7DZCSDLEW7WLRRA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.815712347 (590h28m48.686795382s) [noise=true , address=145.79.227.22:4510 , piece_id=KF4ZTUPQ35KOU3DZ5QGWWYJMEMSRQI7COZIXUMCRMG457YE3IY7Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.816537719 (590h28m48.687620784s) [noise=true , address=77.90.31.15:4697 , piece_id=YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.817015598 (590h28m48.688098633s) [noise=true , address=145.79.227.49:4339 , piece_id=OLAR2GVVB4LAAA62PGUCNALLPXDRPA6MJ7N52UKNT4SO4GFV34CA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.817787278 (590h28m48.688870313s) [noise=true , address=145.79.227.9:4367 , piece_id=YMVFQIHUZ72XLLTUWLDE7XK6JBR43EEIZ5Y766CLZBRHJRWX7WSA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.818169556 (590h28m48.689252601s) [noise=true , address=77.90.31.10:4520 , piece_id=L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.819469922 (590h28m48.690552957s) [noise=true , address=145.79.227.48:4303 , piece_id=LH5GUHTSPVJDYI7XBOYD5QAMRBISZXKG2LAKPKDXEERQC7DWQ7JA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.821074578 (590h28m48.692157613s) [noise=true , address=45.91.116.12:4227 , piece_id=7ILRCBWTBV3NNCM2D25QXIDZNPLPN2QUDZRZZPIZL5F3KJQ232MA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.821461765 (590h28m48.69254481s) [noise=true , address=77.90.31.17:4770 , piece_id=QFXOPMSKXJIRJSHKA7R3JBJLGTGPYXJ2NUTGFTWHRHM6XFUW3VHQ, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:50.434782068 (590h28m50.305865113s) [upload=517647]: Next returned nil segment Dec 30 05:57:50.434794843 (590h28m50.305877878s) [upload=517647]: Waiting for error group managing segments... Dec 30 05:57:50.816278389 (590h28m50.687361424s) [noise=true , address=145.79.227.9:4367 , piece_id=YMVFQIHUZ72XLLTUWLDE7XK6JBR43EEIZ5Y766CLZBRHJRWX7WSA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.816522073 (590h28m50.687605108s) [noise=true , address=145.79.227.48:4303 , piece_id=LH5GUHTSPVJDYI7XBOYD5QAMRBISZXKG2LAKPKDXEERQC7DWQ7JA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.819180385 (590h28m50.69026342s) [noise=true , address=145.79.227.11:4729 , piece_id=WSENCZD6EVDNA5JZGINYTMXF27YXTPO5OEUPSLKJBLFKIJNWTFAA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.819709913 (590h28m50.690792958s) [noise=true , address=145.79.227.16:4911 , piece_id=HJTB3YGOURTBTT577QMKJBBFLWFMPY6NGTXEDKQ4XFOAF2RI6C7A, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.820513734 (590h28m50.691596769s) [noise=true , address=145.79.227.19:4077 , piece_id=KWNDJ2LU2DJG2TLSTU4WVQ7HV6FZPVBQEURJSYWQKBXJ7RQ5DU6Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.821350417 (590h28m50.692433462s) [noise=true , address=145.79.227.7:4294 , piece_id=DKWQOKXHNYISIZ4ND4WWKZD5ECVRDSK55J2SZ7DZCSDLEW7WLRRA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.824387480 (590h28m50.695470515s) [noise=true , address=145.79.227.22:4510 , piece_id=KF4ZTUPQ35KOU3DZ5QGWWYJMEMSRQI7COZIXUMCRMG457YE3IY7Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.892679318 (590h28m50.763762353s) [noise=true , address=145.79.227.38:4977 , piece_id=OONKWQ5VMVSQE2WU4WD62S7MG2AOE3NZQLX7IP22J4OB2ESFTVBA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.999047852 (590h28m50.870130887s) [noise=true , address=77.90.31.10:4520 , piece_id=L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.091779157 (590h28m50.962862192s) [noise=true , address=77.90.31.17:4770 , piece_id=QFXOPMSKXJIRJSHKA7R3JBJLGTGPYXJ2NUTGFTWHRHM6XFUW3VHQ, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.132348578 (590h28m51.003431613s) [noise=true , address=145.79.227.49:4339 , piece_id=OLAR2GVVB4LAAA62PGUCNALLPXDRPA6MJ7N52UKNT4SO4GFV34CA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.164327811 (590h28m51.035410846s) [noise=true , address=77.90.31.15:4697 , piece_id=YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.223011665 (590h28m51.09409471s) [noise=true , address=45.91.116.12:4227 , piece_id=7ILRCBWTBV3NNCM2D25QXIDZNPLPN2QUDZRZZPIZL5F3KJQ232MA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.223037134 (590h28m51.094120179s) [seg_pos={0 0}, upload=517647]: Segment reached optimal threshold of 13 pieces. Dec 30 05:57:51.223255058 (590h28m51.094338093s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 30 05:57:51.223265369 (590h28m51.094348414s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223377812 (590h28m51.094460858s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 05:57:51.223390166 (590h28m51.094473211s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223420163 (590h28m51.094503208s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection Dec 30 05:57:51.223428138 (590h28m51.094511173s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223450210 (590h28m51.094533245s) [seg_pos={0 0}, upload=517647]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:57:51.223530653 (590h28m51.094613698s) [upload=517647]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:57:51.223536685 (590h28m51.094619741s) [upload=517647]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:57:51.515111610 (590h28m51.386194646s) [upload=517647]: Done uploading segments... Dec 30 05:57:51.515157619 (590h28m51.386240664s) [upload=517647]: Upload finished. err: <nil> Duration: 5.817643786s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.68443998s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:58:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.684468596s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XXXXX..X..X...X....X.X.......X..X..X....X.XX.X..X..X..X..XX] duration: 30.00280228s trace: https://jaeger.dev.storj.io/trace/51c87eb35646907f FAIL (2025-12-30 05:58:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA) context deadline exceeded ======LOGS====== c 30 05:57:48.567806875 (590h28m48.43888992s) [upload=517647]: Starting upload Dec 30 05:57:48.567912356 (590h28m48.438995391s) [upload=517647]: Done starting upload Dec 30 05:57:48.567984994 (590h28m48.439068029s) [upload=517647]: Uploading segments... Dec 30 05:57:48.568173282 (590h28m48.439256327s) [upload=517647]: Got next segment. Inline: false Dec 30 05:57:48.568223909 (590h28m48.439306944s) [upload=517647]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:57:48.568233337 (590h28m48.439316382s) [upload=517647]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:57:48.807489480 (590h28m48.678572515s) [seg_pos={0 0}, upload=517647]: Begin upload segment... Dec 30 05:57:48.807529977 (590h28m48.678613012s) [seg_pos={0 0}, upload=517647]: Done begin upload segment. Dec 30 05:57:48.807553841 (590h28m48.678636876s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.811525425 (590h28m48.68260846s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.811564219 (590h28m48.682647264s) [noise=true , address=145.79.227.16:4911 , piece_id=HJTB3YGOURTBTT577QMKJBBFLWFMPY6NGTXEDKQ4XFOAF2RI6C7A, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.812873170 (590h28m48.683956206s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.812910992 (590h28m48.683994037s) [noise=true , address=145.79.227.11:4729 , piece_id=WSENCZD6EVDNA5JZGINYTMXF27YXTPO5OEUPSLKJBLFKIJNWTFAA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.814140935 (590h28m48.68522398s) [noise=true , address=145.79.227.19:4077 , piece_id=KWNDJ2LU2DJG2TLSTU4WVQ7HV6FZPVBQEURJSYWQKBXJ7RQ5DU6Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.814622252 (590h28m48.685705287s) [noise=true , address=145.79.227.38:4977 , piece_id=OONKWQ5VMVSQE2WU4WD62S7MG2AOE3NZQLX7IP22J4OB2ESFTVBA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.815333826 (590h28m48.686416861s) [noise=true , address=145.79.227.7:4294 , piece_id=DKWQOKXHNYISIZ4ND4WWKZD5ECVRDSK55J2SZ7DZCSDLEW7WLRRA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.815712347 (590h28m48.686795382s) [noise=true , address=145.79.227.22:4510 , piece_id=KF4ZTUPQ35KOU3DZ5QGWWYJMEMSRQI7COZIXUMCRMG457YE3IY7Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.816537719 (590h28m48.687620784s) [noise=true , address=77.90.31.15:4697 , piece_id=YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.817015598 (590h28m48.688098633s) [noise=true , address=145.79.227.49:4339 , piece_id=OLAR2GVVB4LAAA62PGUCNALLPXDRPA6MJ7N52UKNT4SO4GFV34CA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.817787278 (590h28m48.688870313s) [noise=true , address=145.79.227.9:4367 , piece_id=YMVFQIHUZ72XLLTUWLDE7XK6JBR43EEIZ5Y766CLZBRHJRWX7WSA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.818169556 (590h28m48.689252601s) [noise=true , address=77.90.31.10:4520 , piece_id=L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.819469922 (590h28m48.690552957s) [noise=true , address=145.79.227.48:4303 , piece_id=LH5GUHTSPVJDYI7XBOYD5QAMRBISZXKG2LAKPKDXEERQC7DWQ7JA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.821074578 (590h28m48.692157613s) [noise=true , address=45.91.116.12:4227 , piece_id=7ILRCBWTBV3NNCM2D25QXIDZNPLPN2QUDZRZZPIZL5F3KJQ232MA, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:48.821461765 (590h28m48.69254481s) [noise=true , address=77.90.31.17:4770 , piece_id=QFXOPMSKXJIRJSHKA7R3JBJLGTGPYXJ2NUTGFTWHRHM6XFUW3VHQ, seg_pos={0 0}, upload=517647]: Uploading piece... Dec 30 05:57:50.434782068 (590h28m50.305865113s) [upload=517647]: Next returned nil segment Dec 30 05:57:50.434794843 (590h28m50.305877878s) [upload=517647]: Waiting for error group managing segments... Dec 30 05:57:50.816278389 (590h28m50.687361424s) [noise=true , address=145.79.227.9:4367 , piece_id=YMVFQIHUZ72XLLTUWLDE7XK6JBR43EEIZ5Y766CLZBRHJRWX7WSA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.816522073 (590h28m50.687605108s) [noise=true , address=145.79.227.48:4303 , piece_id=LH5GUHTSPVJDYI7XBOYD5QAMRBISZXKG2LAKPKDXEERQC7DWQ7JA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.819180385 (590h28m50.69026342s) [noise=true , address=145.79.227.11:4729 , piece_id=WSENCZD6EVDNA5JZGINYTMXF27YXTPO5OEUPSLKJBLFKIJNWTFAA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.819709913 (590h28m50.690792958s) [noise=true , address=145.79.227.16:4911 , piece_id=HJTB3YGOURTBTT577QMKJBBFLWFMPY6NGTXEDKQ4XFOAF2RI6C7A, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.820513734 (590h28m50.691596769s) [noise=true , address=145.79.227.19:4077 , piece_id=KWNDJ2LU2DJG2TLSTU4WVQ7HV6FZPVBQEURJSYWQKBXJ7RQ5DU6Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.821350417 (590h28m50.692433462s) [noise=true , address=145.79.227.7:4294 , piece_id=DKWQOKXHNYISIZ4ND4WWKZD5ECVRDSK55J2SZ7DZCSDLEW7WLRRA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.824387480 (590h28m50.695470515s) [noise=true , address=145.79.227.22:4510 , piece_id=KF4ZTUPQ35KOU3DZ5QGWWYJMEMSRQI7COZIXUMCRMG457YE3IY7Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.892679318 (590h28m50.763762353s) [noise=true , address=145.79.227.38:4977 , piece_id=OONKWQ5VMVSQE2WU4WD62S7MG2AOE3NZQLX7IP22J4OB2ESFTVBA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:50.999047852 (590h28m50.870130887s) [noise=true , address=77.90.31.10:4520 , piece_id=L7PEJK3LU43UDQOFSMVQSJ7WPDXCU2GRHBNA67M2TB47UWEKQ4EA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.091779157 (590h28m50.962862192s) [noise=true , address=77.90.31.17:4770 , piece_id=QFXOPMSKXJIRJSHKA7R3JBJLGTGPYXJ2NUTGFTWHRHM6XFUW3VHQ, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.132348578 (590h28m51.003431613s) [noise=true , address=145.79.227.49:4339 , piece_id=OLAR2GVVB4LAAA62PGUCNALLPXDRPA6MJ7N52UKNT4SO4GFV34CA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.164327811 (590h28m51.035410846s) [noise=true , address=77.90.31.15:4697 , piece_id=YSDRBHGKLE6DY2GPQJDHOLTDVHKAF43JBA5WIR7AHNUUL7BKEL5Q, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.223011665 (590h28m51.09409471s) [noise=true , address=45.91.116.12:4227 , piece_id=7ILRCBWTBV3NNCM2D25QXIDZNPLPN2QUDZRZZPIZL5F3KJQ232MA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: <nil> Dec 30 05:57:51.223037134 (590h28m51.094120179s) [seg_pos={0 0}, upload=517647]: Segment reached optimal threshold of 13 pieces. Dec 30 05:57:51.223255058 (590h28m51.094338093s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF Dec 30 05:57:51.223265369 (590h28m51.094348414s) [noise=true , address=45.91.116.14:4190 , piece_id=LVOQUAFSJQAZMCQ5S7FFQ2LOR6K23GLSMJILMXO5FIDCWGXAB7SA, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223377812 (590h28m51.094460858s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 30 05:57:51.223390166 (590h28m51.094473211s) [noise=true , address=45.91.116.6:4975 , piece_id=QJRQMUX2HFJW2EH2ZF4YITJOHKYTT4IY4N6IP3ZUSF5BSI57NG3A, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223420163 (590h28m51.094503208s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection; write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60480->77.90.31.14:4661: use of closed network connection Dec 30 05:57:51.223428138 (590h28m51.094511173s) [noise=true , address=77.90.31.14:4661 , piece_id=DM34UGXJKL6U4LPYGJSGR3BOCEFJHJRD6D3WN6SE6XPWJ3YSRHQQ, seg_pos={0 0}, upload=517647]: Long tail cancellation due to optimal threshold Dec 30 05:57:51.223450210 (590h28m51.094533245s) [seg_pos={0 0}, upload=517647]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:57:51.223530653 (590h28m51.094613698s) [upload=517647]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:57:51.223536685 (590h28m51.094619741s) [upload=517647]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:57:51.515111610 (590h28m51.386194646s) [upload=517647]: Done uploading segments... Dec 30 05:57:51.515157619 (590h28m51.386240664s) [upload=517647]: Upload finished. err: <nil> Duration: 30.002822888s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 5.001414823s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:58:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001438458s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 4.27379071s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:57:45 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.273842268s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.XXXXX..X..X...X....X.X.......X..X..X....X.XX.X..X..X..X..X] duration: 30.002705736s trace: https://jaeger.dev.storj.io/trace/145732ac690f7b28 FAIL (2025-12-30 05:57:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: 4EBFQU72U4HU4E2GIDU7FE2DCILKRP2CHJXC5O5L66TVCJTGGALA) context deadline exceeded; piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: 4CDQGPUMLCLP4RQTD73TQWP3PV5MHV66D3FZL5EVAOAU6B6QMVWA) context deadline exceeded ======LOGS====== ec 30 05:56:51.293243225 (590h27m51.16432626s) [upload=517627]: Starting upload Dec 30 05:56:51.293292851 (590h27m51.164375886s) [upload=517627]: Done starting upload Dec 30 05:56:51.293375187 (590h27m51.164458242s) [upload=517627]: Uploading segments... Dec 30 05:56:51.293479806 (590h27m51.164562841s) [upload=517627]: Got next segment. Inline: false Dec 30 05:56:51.293520193 (590h27m51.164603228s) [upload=517627]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:56:51.293524311 (590h27m51.164607356s) [upload=517627]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:56:51.539908790 (590h27m51.410991825s) [seg_pos={0 0}, upload=517627]: Begin upload segment... Dec 30 05:56:51.539957663 (590h27m51.411040698s) [seg_pos={0 0}, upload=517627]: Done begin upload segment. Dec 30 05:56:51.539982811 (590h27m51.411065856s) [noise=true , address=145.79.227.16:4912 , piece_id=QYRGCNYQWUWNNAH6HF34P2PMKOGWG7FEA4JUMF7V23WXIIYYMT5Q, seg_pos={0 0}, upload=517627]: Uploading piece... Dec 30 05:56:51.540160630 (590h27m51.411243675s) [noise=true , address=77.90.31.15:4700 , piece_id=4EBFQU72U4HU4E2GIDU7FE2DCILKRP2CHJXC5O5L66TVCJTGGALA, seg_pos={0 0}, upload=517627]: Uploading piece... Dec 30 05:56:51.540921208 (590h27m51.412004253s) [noise=true , address=45.91.116.14:4191 , piece_id=IZW2YO7FCNJGTOL6KIJRV43KI2SCVRLWA3CP7LK7GKG6XP4UL4WQ, seg_pos={0 0}, upload=517627]: Uploading piece... Dec 30 05:56:51.541171755 (590h27m51.41225479s) [noise=true , address=145.79.227.41:4087 , piece_id=5SKPDN7D2PN4PRO5ETSP4CVCVOXJB2B7BR67CZIBMG45TXVYED7A, seg_pos={0 0}, upload=517627]: Uploading piece... Dec 30 05:56:51.542487090 (590h27m51.413570145s) [noise=true , address=77.90.31.10:4517 , piece_id=QGEUJNWPLHACZI5HJZ4FXTATU45U3I7QJTLFMOKSBEAAE2GVU33A, seg_pos={0 0}, upload=517627]: Uploading piece... Dec 30 05:56:51.543100567 (590h27m51.414183612s) [noise=true , address=145.79.227.4:4185 , piece_id=IFLCJT2U5JYNY4TWTURIDIECA3IADQCFEQMQHUY3F4C3N3GFUKMQ, seg_pos={0 0}, upload=517627]: Uploading piece... Dec 30 05:56:51.544689573 (590h27m51.415772619s) [noise=true , address=77.90.31.19:4841 , piece_id=BGEKXOIXH63ZNFHXMQ3DWDMJ67EFJHRW42ZF5VS3XGCTZBVHNKPQ, seg_pos={0 0}, upload=517627]: Uploading piece... Dec 30 05:56:51.546168360 (590h27m51.417251405s) [noise=true , address=145.79.227.12:4767 , piece_id=3HURAXUAOLYCFIBWOEAP6BKXO5PUH5XNR7YBVYQWWCTEU7FJJHHA, seg_pos={0 0}, upload=517627]: Uploading piece... Dec 30 05:56:51.547870099 (590h27m51.418953144s) [noise=true , address=145.79.227.50:4374 , piece_id=J65A2WJ3CQRNLKVCTOO4N45HZHX2ZO4A6LYNUZYOAVYXFDYCDGIA, seg_pos={0 0}, upload=517627]: Uploading piece... Dec 30 05:56:51.548578118 (590h27m51.419661163s) [noise=true , address=145.79.227.25:4619 , piece_id=IMXRWWXZSVXKRKCBCHPESTDGZDRGTNKKN22YOEQIX7LBAG3XJHTA, seg_pos={0 0}, upload=517627]: Uploading piece... Dec 30 05:56:51.549381348 (590h27m51.420464393s) [noise=true , address=145.79.227.33:4870 , piece_id=TSKCFMTTFOMIBGFTR3437BU5FGN7JTFEX2M72H6BN7UAZBYADCMQ, seg_pos={0 0}, upload=517627]: Uploading piece... Dec 30 05:56:51.550403444 (590h27m51.421486489s) [noise=true , address=145.79.227.2:4115 , piece_id=A72MG6LNWJFXQ62MW2Z6DDQZ5TMHXBT6ERBMGXQVQM3IPIMSFW6Q, seg_pos={0 0}, upload=517627]: Uploading piece... Dec 30 05:56:51.550834816 (590h27m51.421917851s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ3NXXRWPEZGSJVPEVSGBH7UQ2EDITWCFJJ2VX5HLCPQE76EHPPQ, seg_pos={0 0}, upload=517627]: Uploading piece... Dec 30 05:56:51.551282106 (590h27m51.422365151s) [noise=true , address=145.79.227.31:4799 , piece_id=YKEXZMHZ66PJIFCCOFS2LPL7JKWWT6UZCJ5CQ236ELPX6ORMQGOQ, seg_pos={0 0}, upload=517627]: Uploading piece... Dec 30 05:56:51.551335909 (590h27m51.422418954s) [noise=true , address=45.91.116.5:4297 , piece_id=4CDQGPUMLCLP4RQTD73TQWP3PV5MHV66D3FZL5EVAOAU6B6QMVWA, seg_pos={0 0}, upload=517627]: Uploading piece... Dec 30 05:56:51.552217628 (590h27m51.423300673s) [noise=true , address=145.79.227.11:4732 , piece_id=VIEFDGZHIQCRXQD5A7E6YDVKCFLDLAEYWPDX25GTBEYHXMYOVH5A, seg_pos={0 0}, upload=517627]: Uploading piece... Dec 30 05:56:53.153614445 (590h27m53.0246975s) [upload=517627]: Next returned nil segment Dec 30 05:56:53.153632891 (590h27m53.024715936s) [upload=517627]: Waiting for error group managing segments... Dec 30 05:56:53.540715603 (590h27m53.411798648s) [noise=true , address=145.79.227.41:4087 , piece_id=5SKPDN7D2PN4PRO5ETSP4CVCVOXJB2B7BR67CZIBMG45TXVYED7A, seg_pos={0 0}, upload=517627]: Done uploading piece. err: <nil> Dec 30 05:56:53.558200597 (590h27m53.429283632s) [noise=true , address=145.79.227.16:4912 , piece_id=QYRGCNYQWUWNNAH6HF34P2PMKOGWG7FEA4JUMF7V23WXIIYYMT5Q, seg_pos={0 0}, upload=517627]: Done uploading piece. err: <nil> Dec 30 05:56:53.561173277 (590h27m53.432256323s) [noise=true , address=145.79.227.2:4115 , piece_id=A72MG6LNWJFXQ62MW2Z6DDQZ5TMHXBT6ERBMGXQVQM3IPIMSFW6Q, seg_pos={0 0}, upload=517627]: Done uploading piece. err: <nil> Dec 30 05:56:53.561983020 (590h27m53.433066065s) [noise=true , address=145.79.227.25:4619 , piece_id=IMXRWWXZSVXKRKCBCHPESTDGZDRGTNKKN22YOEQIX7LBAG3XJHTA, seg_pos={0 0}, upload=517627]: Done uploading piece. err: <nil> Dec 30 05:56:53.567291638 (590h27m53.438374743s) [noise=true , address=145.79.227.31:4799 , piece_id=YKEXZMHZ66PJIFCCOFS2LPL7JKWWT6UZCJ5CQ236ELPX6ORMQGOQ, seg_pos={0 0}, upload=517627]: Done uploading piece. err: <nil> Dec 30 05:56:53.574669647 (590h27m53.445752682s) [noise=true , address=145.79.227.11:4732 , piece_id=VIEFDGZHIQCRXQD5A7E6YDVKCFLDLAEYWPDX25GTBEYHXMYOVH5A, seg_pos={0 0}, upload=517627]: Done uploading piece. err: <nil> Dec 30 05:56:53.625338440 (590h27m53.496421485s) [noise=true , address=145.79.227.12:4767 , piece_id=3HURAXUAOLYCFIBWOEAP6BKXO5PUH5XNR7YBVYQWWCTEU7FJJHHA, seg_pos={0 0}, upload=517627]: Done uploading piece. err: <nil> Dec 30 05:56:53.697014513 (590h27m53.568097548s) [noise=true , address=145.79.227.4:4185 , piece_id=IFLCJT2U5JYNY4TWTURIDIECA3IADQCFEQMQHUY3F4C3N3GFUKMQ, seg_pos={0 0}, upload=517627]: Done uploading piece. err: <nil> Dec 30 05:56:53.697086440 (590h27m53.568169485s) [noise=true , address=145.79.227.50:4374 , piece_id=J65A2WJ3CQRNLKVCTOO4N45HZHX2ZO4A6LYNUZYOAVYXFDYCDGIA, seg_pos={0 0}, upload=517627]: Done uploading piece. err: <nil> Dec 30 05:56:53.804354468 (590h27m53.675437513s) [noise=true , address=77.90.31.21:4878 , piece_id=GJ3NXXRWPEZGSJVPEVSGBH7UQ2EDITWCFJJ2VX5HLCPQE76EHPPQ, seg_pos={0 0}, upload=517627]: Done uploading piece. err: <nil> Dec 30 05:56:53.826193342 (590h27m53.697276377s) [noise=true , address=77.90.31.15:4700 , piece_id=4EBFQU72U4HU4E2GIDU7FE2DCILKRP2CHJXC5O5L66TVCJTGGALA, seg_pos={0 0}, upload=517627]: Done uploading piece. err: <nil> Dec 30 05:56:53.846709979 (590h27m53.717793014s) [noise=true , address=77.90.31.10:4517 , piece_id=QGEUJNWPLHACZI5HJZ4FXTATU45U3I7QJTLFMOKSBEAAE2GVU33A, seg_pos={0 0}, upload=517627]: Done uploading piece. err: <nil> Dec 30 05:56:53.989698454 (590h27m53.860781499s) [noise=true , address=45.91.116.5:4297 , piece_id=4CDQGPUMLCLP4RQTD73TQWP3PV5MHV66D3FZL5EVAOAU6B6QMVWA, seg_pos={0 0}, upload=517627]: Done uploading piece. err: <nil> Dec 30 05:56:53.989732599 (590h27m53.860815644s) [seg_pos={0 0}, upload=517627]: Segment reached optimal threshold of 13 pieces. Dec 30 05:56:53.990038632 (590h27m53.861121677s) [noise=true , address=77.90.31.19:4841 , piece_id=BGEKXOIXH63ZNFHXMQ3DWDMJ67EFJHRW42ZF5VS3XGCTZBVHNKPQ, seg_pos={0 0}, upload=517627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:38704->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:38704->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38704->77.90.31.19:4841: use of closed network connection Dec 30 05:56:53.990048551 (590h27m53.861131606s) [noise=true , address=77.90.31.19:4841 , piece_id=BGEKXOIXH63ZNFHXMQ3DWDMJ67EFJHRW42ZF5VS3XGCTZBVHNKPQ, seg_pos={0 0}, upload=517627]: Long tail cancellation due to optimal threshold Dec 30 05:56:53.990456117 (590h27m53.861539162s) [noise=true , address=145.79.227.33:4870 , piece_id=TSKCFMTTFOMIBGFTR3437BU5FGN7JTFEX2M72H6BN7UAZBYADCMQ, seg_pos={0 0}, upload=517627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 30 05:56:53.990474652 (590h27m53.861557697s) [noise=true , address=145.79.227.33:4870 , piece_id=TSKCFMTTFOMIBGFTR3437BU5FGN7JTFEX2M72H6BN7UAZBYADCMQ, seg_pos={0 0}, upload=517627]: Long tail cancellation due to optimal threshold Dec 30 05:56:53.990537151 (590h27m53.861620196s) [noise=true , address=45.91.116.14:4191 , piece_id=IZW2YO7FCNJGTOL6KIJRV43KI2SCVRLWA3CP7LK7GKG6XP4UL4WQ, seg_pos={0 0}, upload=517627]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 30 05:56:53.990548382 (590h27m53.861631427s) [noise=true , address=45.91.116.14:4191 , piece_id=IZW2YO7FCNJGTOL6KIJRV43KI2SCVRLWA3CP7LK7GKG6XP4UL4WQ, seg_pos={0 0}, upload=517627]: Long tail cancellation due to optimal threshold Dec 30 05:56:53.990674984 (590h27m53.861758029s) [seg_pos={0 0}, upload=517627]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:56:53.990767731 (590h27m53.861850776s) [upload=517627]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:56:53.990775615 (590h27m53.86185866s) [upload=517627]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:56:54.278733497 (590h27m54.149816532s) [upload=517627]: Done uploading segments... Dec 30 05:56:54.278753285 (590h27m54.14983633s) [upload=517627]: Upload finished. err: <nil> Duration: 30.002726455s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.499399693s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:57:19 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.499417457s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [X........X..................................................] duration: 3.492880792s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:56:37 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.492905089s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X.XXXXX..X..X...X....X.X.......X..X..X....X.XX.X..X..X..X..] duration: 5.901611742s trace: https://jaeger.dev.storj.io/trace/5c754efab9c97c3f FAIL (2025-12-30 05:56:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 05:54:57.896027623 (590h25m57.767110658s) [upload=517599]: Starting upload Dec 30 05:54:57.896114809 (590h25m57.767197844s) [upload=517599]: Done starting upload Dec 30 05:54:57.896179262 (590h25m57.767262307s) [upload=517599]: Uploading segments... Dec 30 05:54:57.896341010 (590h25m57.767424045s) [upload=517599]: Got next segment. Inline: false Dec 30 05:54:57.896374935 (590h25m57.76745797s) [upload=517599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:54:57.896379223 (590h25m57.767462278s) [upload=517599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:54:58.143259265 (590h25m58.0143423s) [seg_pos={0 0}, upload=517599]: Begin upload segment... Dec 30 05:54:58.143300473 (590h25m58.014383508s) [seg_pos={0 0}, upload=517599]: Done begin upload segment. Dec 30 05:54:58.143326343 (590h25m58.014409378s) [noise=true , address=45.91.116.10:4084 , piece_id=C3INDBKAT27LA5GVKSFVHIIXVY7UBYSH7L2C3WVFJ3QLKKM4H3QQ, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.143404752 (590h25m58.014487797s) [noise=true , address=145.79.227.12:4765 , piece_id=Q7AI53PGNNPVK32SVXHWUEACF44NDJJSMMUF7VUEV77MT6XKHMDA, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.144016327 (590h25m58.015099372s) [noise=true , address=145.79.227.15:4875 , piece_id=XTU2T4QD2TBHTSMM75I7TFFDGNGUZ2NFD6SS2M2767NELN52ALCA, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.144502392 (590h25m58.015585437s) [noise=true , address=77.90.31.11:4554 , piece_id=FNYJDKOZXQPG54IVDPRZJU2JUIFKLX7Z6RFDZBYDKDKHW57K4WEQ, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.144937560 (590h25m58.016020605s) [noise=true , address=145.79.227.22:4509 , piece_id=SJFXPOXQKZGGMJFGWHVFKFRRZSJJDSWF63JWQJ6RIJHCZFJUX4JQ, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.145440599 (590h25m58.016523644s) [noise=true , address=45.91.116.11:4117 , piece_id=KSXK4YJOBPRVFTDVQP3M37PGQUZAX6H4SG2FNKM5E6BI5ZFUSRDA, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.145952563 (590h25m58.017035618s) [noise=true , address=77.90.31.15:4700 , piece_id=GJ2QGNWHFS4M2EP372RQDU34DK5PWDHK42TONUYP333Z4H6CVZWA, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.146924414 (590h25m58.018007449s) [noise=true , address=145.79.227.38:4977 , piece_id=356XNYVVHHMEXRX2M7GRI2RSB2MAHJOBT5ZGJGOOEPCUW5R4ND3A, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.147846901 (590h25m58.018929936s) [noise=true , address=145.79.227.6:4258 , piece_id=GVQLRQOYQ5CG5M2ORAPS3BNSQR6XRAXZMMNXKJXJ5CTYOXZBYNHQ, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.147987919 (590h25m58.019070954s) [noise=true , address=77.90.31.10:4519 , piece_id=SDKQIMB5G6A7KAA5TRIHVTLR75WDORTGCT6YCGPDKLDEABISA3WA, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.149100007 (590h25m58.020183042s) [noise=true , address=145.79.227.31:4799 , piece_id=5H5HAOZ2FKLCPL2B6OI2NZNJGJS4UAOYQEWWK6YMFTMWPYMQPTHA, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.149733573 (590h25m58.020816608s) [noise=true , address=77.90.31.14:4661 , piece_id=HTPSJRHPVAOEM4R4D7I6OEVOAPSAQSQHGTUN2DOHYCM76XHAJ64A, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.150212816 (590h25m58.021295851s) [noise=true , address=145.79.227.46:4601 , piece_id=2LXFVVSRO5RO3CEJWHR3TB5GM7SH7ACFPLGR5OSEXGU7AES2VGXA, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.151500858 (590h25m58.022583893s) [noise=true , address=145.79.227.18:4006 , piece_id=HLMBZMG24VGL63TBOROH36NGOU4U2OHDBG25DZ7YI3B6QWVUBAMQ, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.152664634 (590h25m58.023747669s) [noise=true , address=145.79.227.21:4473 , piece_id=RXLO4TEXJR7DS266EIGLSW4VBKWW6TXIFVP5VZWBYNKO3IQRNMJA, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.153772544 (590h25m58.024855579s) [noise=true , address=145.79.227.52:4446 , piece_id=TSYEI56FRX6LFQL5ZYLBP2YU7U7JDFLIBQEBJDFSEDF56KPPB6WQ, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:59.857460631 (590h25m59.728543666s) [upload=517599]: Next returned nil segment Dec 30 05:54:59.857474728 (590h25m59.728557763s) [upload=517599]: Waiting for error group managing segments... Dec 30 05:55:00.140898225 (590h26m0.011981271s) [noise=true , address=145.79.227.38:4977 , piece_id=356XNYVVHHMEXRX2M7GRI2RSB2MAHJOBT5ZGJGOOEPCUW5R4ND3A, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.144002196 (590h26m0.015085231s) [noise=true , address=145.79.227.6:4258 , piece_id=GVQLRQOYQ5CG5M2ORAPS3BNSQR6XRAXZMMNXKJXJ5CTYOXZBYNHQ, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.147402932 (590h26m0.018485967s) [noise=true , address=145.79.227.46:4601 , piece_id=2LXFVVSRO5RO3CEJWHR3TB5GM7SH7ACFPLGR5OSEXGU7AES2VGXA, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.151078601 (590h26m0.022161636s) [noise=true , address=145.79.227.52:4446 , piece_id=TSYEI56FRX6LFQL5ZYLBP2YU7U7JDFLIBQEBJDFSEDF56KPPB6WQ, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.155973531 (590h26m0.027056566s) [noise=true , address=145.79.227.22:4509 , piece_id=SJFXPOXQKZGGMJFGWHVFKFRRZSJJDSWF63JWQJ6RIJHCZFJUX4JQ, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.161855371 (590h26m0.032938407s) [noise=true , address=145.79.227.15:4875 , piece_id=XTU2T4QD2TBHTSMM75I7TFFDGNGUZ2NFD6SS2M2767NELN52ALCA, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.165212274 (590h26m0.036295309s) [noise=true , address=145.79.227.31:4799 , piece_id=5H5HAOZ2FKLCPL2B6OI2NZNJGJS4UAOYQEWWK6YMFTMWPYMQPTHA, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.167560976 (590h26m0.038644011s) [noise=true , address=145.79.227.18:4006 , piece_id=HLMBZMG24VGL63TBOROH36NGOU4U2OHDBG25DZ7YI3B6QWVUBAMQ, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.167621021 (590h26m0.038704056s) [noise=true , address=145.79.227.21:4473 , piece_id=RXLO4TEXJR7DS266EIGLSW4VBKWW6TXIFVP5VZWBYNKO3IQRNMJA, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.177931353 (590h26m0.049014388s) [noise=true , address=145.79.227.12:4765 , piece_id=Q7AI53PGNNPVK32SVXHWUEACF44NDJJSMMUF7VUEV77MT6XKHMDA, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.337525959 (590h26m0.208609004s) [noise=true , address=77.90.31.15:4700 , piece_id=GJ2QGNWHFS4M2EP372RQDU34DK5PWDHK42TONUYP333Z4H6CVZWA, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.436954538 (590h26m0.308037573s) [noise=true , address=77.90.31.14:4661 , piece_id=HTPSJRHPVAOEM4R4D7I6OEVOAPSAQSQHGTUN2DOHYCM76XHAJ64A, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.565785264 (590h26m0.436868309s) [noise=true , address=45.91.116.11:4117 , piece_id=KSXK4YJOBPRVFTDVQP3M37PGQUZAX6H4SG2FNKM5E6BI5ZFUSRDA, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.565843866 (590h26m0.436926921s) [seg_pos={0 0}, upload=517599]: Segment reached optimal threshold of 13 pieces. Dec 30 05:55:00.566213469 (590h26m0.437296504s) [noise=true , address=77.90.31.11:4554 , piece_id=FNYJDKOZXQPG54IVDPRZJU2JUIFKLX7Z6RFDZBYDKDKHW57K4WEQ, seg_pos={0 0}, upload=517599]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 30 05:55:00.566296638 (590h26m0.437379683s) [noise=true , address=77.90.31.11:4554 , piece_id=FNYJDKOZXQPG54IVDPRZJU2JUIFKLX7Z6RFDZBYDKDKHW57K4WEQ, seg_pos={0 0}, upload=517599]: Long tail cancellation due to optimal threshold Dec 30 05:55:00.566383012 (590h26m0.437466057s) [noise=true , address=77.90.31.10:4519 , piece_id=SDKQIMB5G6A7KAA5TRIHVTLR75WDORTGCT6YCGPDKLDEABISA3WA, seg_pos={0 0}, upload=517599]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 30 05:55:00.566390646 (590h26m0.437473692s) [noise=true , address=77.90.31.10:4519 , piece_id=SDKQIMB5G6A7KAA5TRIHVTLR75WDORTGCT6YCGPDKLDEABISA3WA, seg_pos={0 0}, upload=517599]: Long tail cancellation due to optimal threshold Dec 30 05:55:00.566406978 (590h26m0.437490013s) [noise=true , address=45.91.116.10:4084 , piece_id=C3INDBKAT27LA5GVKSFVHIIXVY7UBYSH7L2C3WVFJ3QLKKM4H3QQ, seg_pos={0 0}, upload=517599]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 30 05:55:00.566414522 (590h26m0.437497567s) [noise=true , address=45.91.116.10:4084 , piece_id=C3INDBKAT27LA5GVKSFVHIIXVY7UBYSH7L2C3WVFJ3QLKKM4H3QQ, seg_pos={0 0}, upload=517599]: Long tail cancellation due to optimal threshold Dec 30 05:55:00.566439981 (590h26m0.437523016s) [seg_pos={0 0}, upload=517599]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:55:00.566521426 (590h26m0.437604471s) [upload=517599]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:55:00.566527788 (590h26m0.437610823s) [upload=517599]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:55:00.852212708 (590h26m0.723295743s) [upload=517599]: Done uploading segments... Dec 30 05:55:00.852231875 (590h26m0.72331491s) [upload=517599]: Upload finished. err: <nil> Duration: 5.901643182s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..X.XXXXX..X..X...X....X.X.......X..X..X....X.XX.X..X..X..X.] duration: 5.780152884s trace: https://jaeger.dev.storj.io/trace/5c754efab9c97c3f FAIL (2025-12-30 05:55:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 05:54:57.896027623 (590h25m57.767110658s) [upload=517599]: Starting upload Dec 30 05:54:57.896114809 (590h25m57.767197844s) [upload=517599]: Done starting upload Dec 30 05:54:57.896179262 (590h25m57.767262307s) [upload=517599]: Uploading segments... Dec 30 05:54:57.896341010 (590h25m57.767424045s) [upload=517599]: Got next segment. Inline: false Dec 30 05:54:57.896374935 (590h25m57.76745797s) [upload=517599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:54:57.896379223 (590h25m57.767462278s) [upload=517599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:54:58.143259265 (590h25m58.0143423s) [seg_pos={0 0}, upload=517599]: Begin upload segment... Dec 30 05:54:58.143300473 (590h25m58.014383508s) [seg_pos={0 0}, upload=517599]: Done begin upload segment. Dec 30 05:54:58.143326343 (590h25m58.014409378s) [noise=true , address=45.91.116.10:4084 , piece_id=C3INDBKAT27LA5GVKSFVHIIXVY7UBYSH7L2C3WVFJ3QLKKM4H3QQ, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.143404752 (590h25m58.014487797s) [noise=true , address=145.79.227.12:4765 , piece_id=Q7AI53PGNNPVK32SVXHWUEACF44NDJJSMMUF7VUEV77MT6XKHMDA, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.144016327 (590h25m58.015099372s) [noise=true , address=145.79.227.15:4875 , piece_id=XTU2T4QD2TBHTSMM75I7TFFDGNGUZ2NFD6SS2M2767NELN52ALCA, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.144502392 (590h25m58.015585437s) [noise=true , address=77.90.31.11:4554 , piece_id=FNYJDKOZXQPG54IVDPRZJU2JUIFKLX7Z6RFDZBYDKDKHW57K4WEQ, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.144937560 (590h25m58.016020605s) [noise=true , address=145.79.227.22:4509 , piece_id=SJFXPOXQKZGGMJFGWHVFKFRRZSJJDSWF63JWQJ6RIJHCZFJUX4JQ, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.145440599 (590h25m58.016523644s) [noise=true , address=45.91.116.11:4117 , piece_id=KSXK4YJOBPRVFTDVQP3M37PGQUZAX6H4SG2FNKM5E6BI5ZFUSRDA, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.145952563 (590h25m58.017035618s) [noise=true , address=77.90.31.15:4700 , piece_id=GJ2QGNWHFS4M2EP372RQDU34DK5PWDHK42TONUYP333Z4H6CVZWA, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.146924414 (590h25m58.018007449s) [noise=true , address=145.79.227.38:4977 , piece_id=356XNYVVHHMEXRX2M7GRI2RSB2MAHJOBT5ZGJGOOEPCUW5R4ND3A, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.147846901 (590h25m58.018929936s) [noise=true , address=145.79.227.6:4258 , piece_id=GVQLRQOYQ5CG5M2ORAPS3BNSQR6XRAXZMMNXKJXJ5CTYOXZBYNHQ, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.147987919 (590h25m58.019070954s) [noise=true , address=77.90.31.10:4519 , piece_id=SDKQIMB5G6A7KAA5TRIHVTLR75WDORTGCT6YCGPDKLDEABISA3WA, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.149100007 (590h25m58.020183042s) [noise=true , address=145.79.227.31:4799 , piece_id=5H5HAOZ2FKLCPL2B6OI2NZNJGJS4UAOYQEWWK6YMFTMWPYMQPTHA, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.149733573 (590h25m58.020816608s) [noise=true , address=77.90.31.14:4661 , piece_id=HTPSJRHPVAOEM4R4D7I6OEVOAPSAQSQHGTUN2DOHYCM76XHAJ64A, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.150212816 (590h25m58.021295851s) [noise=true , address=145.79.227.46:4601 , piece_id=2LXFVVSRO5RO3CEJWHR3TB5GM7SH7ACFPLGR5OSEXGU7AES2VGXA, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.151500858 (590h25m58.022583893s) [noise=true , address=145.79.227.18:4006 , piece_id=HLMBZMG24VGL63TBOROH36NGOU4U2OHDBG25DZ7YI3B6QWVUBAMQ, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.152664634 (590h25m58.023747669s) [noise=true , address=145.79.227.21:4473 , piece_id=RXLO4TEXJR7DS266EIGLSW4VBKWW6TXIFVP5VZWBYNKO3IQRNMJA, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.153772544 (590h25m58.024855579s) [noise=true , address=145.79.227.52:4446 , piece_id=TSYEI56FRX6LFQL5ZYLBP2YU7U7JDFLIBQEBJDFSEDF56KPPB6WQ, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:59.857460631 (590h25m59.728543666s) [upload=517599]: Next returned nil segment Dec 30 05:54:59.857474728 (590h25m59.728557763s) [upload=517599]: Waiting for error group managing segments... Dec 30 05:55:00.140898225 (590h26m0.011981271s) [noise=true , address=145.79.227.38:4977 , piece_id=356XNYVVHHMEXRX2M7GRI2RSB2MAHJOBT5ZGJGOOEPCUW5R4ND3A, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.144002196 (590h26m0.015085231s) [noise=true , address=145.79.227.6:4258 , piece_id=GVQLRQOYQ5CG5M2ORAPS3BNSQR6XRAXZMMNXKJXJ5CTYOXZBYNHQ, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.147402932 (590h26m0.018485967s) [noise=true , address=145.79.227.46:4601 , piece_id=2LXFVVSRO5RO3CEJWHR3TB5GM7SH7ACFPLGR5OSEXGU7AES2VGXA, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.151078601 (590h26m0.022161636s) [noise=true , address=145.79.227.52:4446 , piece_id=TSYEI56FRX6LFQL5ZYLBP2YU7U7JDFLIBQEBJDFSEDF56KPPB6WQ, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.155973531 (590h26m0.027056566s) [noise=true , address=145.79.227.22:4509 , piece_id=SJFXPOXQKZGGMJFGWHVFKFRRZSJJDSWF63JWQJ6RIJHCZFJUX4JQ, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.161855371 (590h26m0.032938407s) [noise=true , address=145.79.227.15:4875 , piece_id=XTU2T4QD2TBHTSMM75I7TFFDGNGUZ2NFD6SS2M2767NELN52ALCA, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.165212274 (590h26m0.036295309s) [noise=true , address=145.79.227.31:4799 , piece_id=5H5HAOZ2FKLCPL2B6OI2NZNJGJS4UAOYQEWWK6YMFTMWPYMQPTHA, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.167560976 (590h26m0.038644011s) [noise=true , address=145.79.227.18:4006 , piece_id=HLMBZMG24VGL63TBOROH36NGOU4U2OHDBG25DZ7YI3B6QWVUBAMQ, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.167621021 (590h26m0.038704056s) [noise=true , address=145.79.227.21:4473 , piece_id=RXLO4TEXJR7DS266EIGLSW4VBKWW6TXIFVP5VZWBYNKO3IQRNMJA, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.177931353 (590h26m0.049014388s) [noise=true , address=145.79.227.12:4765 , piece_id=Q7AI53PGNNPVK32SVXHWUEACF44NDJJSMMUF7VUEV77MT6XKHMDA, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.337525959 (590h26m0.208609004s) [noise=true , address=77.90.31.15:4700 , piece_id=GJ2QGNWHFS4M2EP372RQDU34DK5PWDHK42TONUYP333Z4H6CVZWA, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.436954538 (590h26m0.308037573s) [noise=true , address=77.90.31.14:4661 , piece_id=HTPSJRHPVAOEM4R4D7I6OEVOAPSAQSQHGTUN2DOHYCM76XHAJ64A, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.565785264 (590h26m0.436868309s) [noise=true , address=45.91.116.11:4117 , piece_id=KSXK4YJOBPRVFTDVQP3M37PGQUZAX6H4SG2FNKM5E6BI5ZFUSRDA, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.565843866 (590h26m0.436926921s) [seg_pos={0 0}, upload=517599]: Segment reached optimal threshold of 13 pieces. Dec 30 05:55:00.566213469 (590h26m0.437296504s) [noise=true , address=77.90.31.11:4554 , piece_id=FNYJDKOZXQPG54IVDPRZJU2JUIFKLX7Z6RFDZBYDKDKHW57K4WEQ, seg_pos={0 0}, upload=517599]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 30 05:55:00.566296638 (590h26m0.437379683s) [noise=true , address=77.90.31.11:4554 , piece_id=FNYJDKOZXQPG54IVDPRZJU2JUIFKLX7Z6RFDZBYDKDKHW57K4WEQ, seg_pos={0 0}, upload=517599]: Long tail cancellation due to optimal threshold Dec 30 05:55:00.566383012 (590h26m0.437466057s) [noise=true , address=77.90.31.10:4519 , piece_id=SDKQIMB5G6A7KAA5TRIHVTLR75WDORTGCT6YCGPDKLDEABISA3WA, seg_pos={0 0}, upload=517599]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 30 05:55:00.566390646 (590h26m0.437473692s) [noise=true , address=77.90.31.10:4519 , piece_id=SDKQIMB5G6A7KAA5TRIHVTLR75WDORTGCT6YCGPDKLDEABISA3WA, seg_pos={0 0}, upload=517599]: Long tail cancellation due to optimal threshold Dec 30 05:55:00.566406978 (590h26m0.437490013s) [noise=true , address=45.91.116.10:4084 , piece_id=C3INDBKAT27LA5GVKSFVHIIXVY7UBYSH7L2C3WVFJ3QLKKM4H3QQ, seg_pos={0 0}, upload=517599]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 30 05:55:00.566414522 (590h26m0.437497567s) [noise=true , address=45.91.116.10:4084 , piece_id=C3INDBKAT27LA5GVKSFVHIIXVY7UBYSH7L2C3WVFJ3QLKKM4H3QQ, seg_pos={0 0}, upload=517599]: Long tail cancellation due to optimal threshold Dec 30 05:55:00.566439981 (590h26m0.437523016s) [seg_pos={0 0}, upload=517599]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:55:00.566521426 (590h26m0.437604471s) [upload=517599]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:55:00.566527788 (590h26m0.437610823s) [upload=517599]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:55:00.852212708 (590h26m0.723295743s) [upload=517599]: Done uploading segments... Dec 30 05:55:00.852231875 (590h26m0.72331491s) [upload=517599]: Upload finished. err: <nil> Duration: 5.780170668s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [...X.XXXXX..X..X...X....X.X.......X..X..X....X.XX.X..X..X..X] duration: 30.003138595s trace: https://jaeger.dev.storj.io/trace/5c754efab9c97c3f FAIL (2025-12-30 05:55:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 05:54:57.896027623 (590h25m57.767110658s) [upload=517599]: Starting upload Dec 30 05:54:57.896114809 (590h25m57.767197844s) [upload=517599]: Done starting upload Dec 30 05:54:57.896179262 (590h25m57.767262307s) [upload=517599]: Uploading segments... Dec 30 05:54:57.896341010 (590h25m57.767424045s) [upload=517599]: Got next segment. Inline: false Dec 30 05:54:57.896374935 (590h25m57.76745797s) [upload=517599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:54:57.896379223 (590h25m57.767462278s) [upload=517599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:54:58.143259265 (590h25m58.0143423s) [seg_pos={0 0}, upload=517599]: Begin upload segment... Dec 30 05:54:58.143300473 (590h25m58.014383508s) [seg_pos={0 0}, upload=517599]: Done begin upload segment. Dec 30 05:54:58.143326343 (590h25m58.014409378s) [noise=true , address=45.91.116.10:4084 , piece_id=C3INDBKAT27LA5GVKSFVHIIXVY7UBYSH7L2C3WVFJ3QLKKM4H3QQ, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.143404752 (590h25m58.014487797s) [noise=true , address=145.79.227.12:4765 , piece_id=Q7AI53PGNNPVK32SVXHWUEACF44NDJJSMMUF7VUEV77MT6XKHMDA, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.144016327 (590h25m58.015099372s) [noise=true , address=145.79.227.15:4875 , piece_id=XTU2T4QD2TBHTSMM75I7TFFDGNGUZ2NFD6SS2M2767NELN52ALCA, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.144502392 (590h25m58.015585437s) [noise=true , address=77.90.31.11:4554 , piece_id=FNYJDKOZXQPG54IVDPRZJU2JUIFKLX7Z6RFDZBYDKDKHW57K4WEQ, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.144937560 (590h25m58.016020605s) [noise=true , address=145.79.227.22:4509 , piece_id=SJFXPOXQKZGGMJFGWHVFKFRRZSJJDSWF63JWQJ6RIJHCZFJUX4JQ, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.145440599 (590h25m58.016523644s) [noise=true , address=45.91.116.11:4117 , piece_id=KSXK4YJOBPRVFTDVQP3M37PGQUZAX6H4SG2FNKM5E6BI5ZFUSRDA, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.145952563 (590h25m58.017035618s) [noise=true , address=77.90.31.15:4700 , piece_id=GJ2QGNWHFS4M2EP372RQDU34DK5PWDHK42TONUYP333Z4H6CVZWA, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.146924414 (590h25m58.018007449s) [noise=true , address=145.79.227.38:4977 , piece_id=356XNYVVHHMEXRX2M7GRI2RSB2MAHJOBT5ZGJGOOEPCUW5R4ND3A, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.147846901 (590h25m58.018929936s) [noise=true , address=145.79.227.6:4258 , piece_id=GVQLRQOYQ5CG5M2ORAPS3BNSQR6XRAXZMMNXKJXJ5CTYOXZBYNHQ, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.147987919 (590h25m58.019070954s) [noise=true , address=77.90.31.10:4519 , piece_id=SDKQIMB5G6A7KAA5TRIHVTLR75WDORTGCT6YCGPDKLDEABISA3WA, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.149100007 (590h25m58.020183042s) [noise=true , address=145.79.227.31:4799 , piece_id=5H5HAOZ2FKLCPL2B6OI2NZNJGJS4UAOYQEWWK6YMFTMWPYMQPTHA, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.149733573 (590h25m58.020816608s) [noise=true , address=77.90.31.14:4661 , piece_id=HTPSJRHPVAOEM4R4D7I6OEVOAPSAQSQHGTUN2DOHYCM76XHAJ64A, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.150212816 (590h25m58.021295851s) [noise=true , address=145.79.227.46:4601 , piece_id=2LXFVVSRO5RO3CEJWHR3TB5GM7SH7ACFPLGR5OSEXGU7AES2VGXA, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.151500858 (590h25m58.022583893s) [noise=true , address=145.79.227.18:4006 , piece_id=HLMBZMG24VGL63TBOROH36NGOU4U2OHDBG25DZ7YI3B6QWVUBAMQ, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.152664634 (590h25m58.023747669s) [noise=true , address=145.79.227.21:4473 , piece_id=RXLO4TEXJR7DS266EIGLSW4VBKWW6TXIFVP5VZWBYNKO3IQRNMJA, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:58.153772544 (590h25m58.024855579s) [noise=true , address=145.79.227.52:4446 , piece_id=TSYEI56FRX6LFQL5ZYLBP2YU7U7JDFLIBQEBJDFSEDF56KPPB6WQ, seg_pos={0 0}, upload=517599]: Uploading piece... Dec 30 05:54:59.857460631 (590h25m59.728543666s) [upload=517599]: Next returned nil segment Dec 30 05:54:59.857474728 (590h25m59.728557763s) [upload=517599]: Waiting for error group managing segments... Dec 30 05:55:00.140898225 (590h26m0.011981271s) [noise=true , address=145.79.227.38:4977 , piece_id=356XNYVVHHMEXRX2M7GRI2RSB2MAHJOBT5ZGJGOOEPCUW5R4ND3A, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.144002196 (590h26m0.015085231s) [noise=true , address=145.79.227.6:4258 , piece_id=GVQLRQOYQ5CG5M2ORAPS3BNSQR6XRAXZMMNXKJXJ5CTYOXZBYNHQ, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.147402932 (590h26m0.018485967s) [noise=true , address=145.79.227.46:4601 , piece_id=2LXFVVSRO5RO3CEJWHR3TB5GM7SH7ACFPLGR5OSEXGU7AES2VGXA, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.151078601 (590h26m0.022161636s) [noise=true , address=145.79.227.52:4446 , piece_id=TSYEI56FRX6LFQL5ZYLBP2YU7U7JDFLIBQEBJDFSEDF56KPPB6WQ, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.155973531 (590h26m0.027056566s) [noise=true , address=145.79.227.22:4509 , piece_id=SJFXPOXQKZGGMJFGWHVFKFRRZSJJDSWF63JWQJ6RIJHCZFJUX4JQ, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.161855371 (590h26m0.032938407s) [noise=true , address=145.79.227.15:4875 , piece_id=XTU2T4QD2TBHTSMM75I7TFFDGNGUZ2NFD6SS2M2767NELN52ALCA, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.165212274 (590h26m0.036295309s) [noise=true , address=145.79.227.31:4799 , piece_id=5H5HAOZ2FKLCPL2B6OI2NZNJGJS4UAOYQEWWK6YMFTMWPYMQPTHA, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.167560976 (590h26m0.038644011s) [noise=true , address=145.79.227.18:4006 , piece_id=HLMBZMG24VGL63TBOROH36NGOU4U2OHDBG25DZ7YI3B6QWVUBAMQ, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.167621021 (590h26m0.038704056s) [noise=true , address=145.79.227.21:4473 , piece_id=RXLO4TEXJR7DS266EIGLSW4VBKWW6TXIFVP5VZWBYNKO3IQRNMJA, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.177931353 (590h26m0.049014388s) [noise=true , address=145.79.227.12:4765 , piece_id=Q7AI53PGNNPVK32SVXHWUEACF44NDJJSMMUF7VUEV77MT6XKHMDA, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.337525959 (590h26m0.208609004s) [noise=true , address=77.90.31.15:4700 , piece_id=GJ2QGNWHFS4M2EP372RQDU34DK5PWDHK42TONUYP333Z4H6CVZWA, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.436954538 (590h26m0.308037573s) [noise=true , address=77.90.31.14:4661 , piece_id=HTPSJRHPVAOEM4R4D7I6OEVOAPSAQSQHGTUN2DOHYCM76XHAJ64A, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.565785264 (590h26m0.436868309s) [noise=true , address=45.91.116.11:4117 , piece_id=KSXK4YJOBPRVFTDVQP3M37PGQUZAX6H4SG2FNKM5E6BI5ZFUSRDA, seg_pos={0 0}, upload=517599]: Done uploading piece. err: <nil> Dec 30 05:55:00.565843866 (590h26m0.436926921s) [seg_pos={0 0}, upload=517599]: Segment reached optimal threshold of 13 pieces. Dec 30 05:55:00.566213469 (590h26m0.437296504s) [noise=true , address=77.90.31.11:4554 , piece_id=FNYJDKOZXQPG54IVDPRZJU2JUIFKLX7Z6RFDZBYDKDKHW57K4WEQ, seg_pos={0 0}, upload=517599]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Dec 30 05:55:00.566296638 (590h26m0.437379683s) [noise=true , address=77.90.31.11:4554 , piece_id=FNYJDKOZXQPG54IVDPRZJU2JUIFKLX7Z6RFDZBYDKDKHW57K4WEQ, seg_pos={0 0}, upload=517599]: Long tail cancellation due to optimal threshold Dec 30 05:55:00.566383012 (590h26m0.437466057s) [noise=true , address=77.90.31.10:4519 , piece_id=SDKQIMB5G6A7KAA5TRIHVTLR75WDORTGCT6YCGPDKLDEABISA3WA, seg_pos={0 0}, upload=517599]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX): protocol: expected piece hash; context canceled; EOF Dec 30 05:55:00.566390646 (590h26m0.437473692s) [noise=true , address=77.90.31.10:4519 , piece_id=SDKQIMB5G6A7KAA5TRIHVTLR75WDORTGCT6YCGPDKLDEABISA3WA, seg_pos={0 0}, upload=517599]: Long tail cancellation due to optimal threshold Dec 30 05:55:00.566406978 (590h26m0.437490013s) [noise=true , address=45.91.116.10:4084 , piece_id=C3INDBKAT27LA5GVKSFVHIIXVY7UBYSH7L2C3WVFJ3QLKKM4H3QQ, seg_pos={0 0}, upload=517599]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 30 05:55:00.566414522 (590h26m0.437497567s) [noise=true , address=45.91.116.10:4084 , piece_id=C3INDBKAT27LA5GVKSFVHIIXVY7UBYSH7L2C3WVFJ3QLKKM4H3QQ, seg_pos={0 0}, upload=517599]: Long tail cancellation due to optimal threshold Dec 30 05:55:00.566439981 (590h26m0.437523016s) [seg_pos={0 0}, upload=517599]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:55:00.566521426 (590h26m0.437604471s) [upload=517599]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:55:00.566527788 (590h26m0.437610823s) [upload=517599]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:55:00.852212708 (590h26m0.723295743s) [upload=517599]: Done uploading segments... Dec 30 05:55:00.852231875 (590h26m0.72331491s) [upload=517599]: Upload finished. err: <nil> Duration: 30.003160727s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.X........X.................................................] duration: 5.001089891s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:55:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001123866s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X...X.XXXXX..X..X...X....X.X.......X..X..X....X.XX.X..X..X..] duration: 6.493402925s trace: https://jaeger.dev.storj.io/trace/18359d6bcf1fdf1f FAIL (2025-12-30 05:54:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: YNLPZ7DXQYKJZSVJDDDGT2GFIMHNMYHWVDB5KWBDOEZZ7V7AWEDA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: XC47E6Z5T3CHAHMXFXWWK2FL6ADOJXBBAAXHDHQYJIINJE7UGUVA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: GKK6HCHFEJSZTZXCMDPRTI533ULZSBRBPKULYEU33E367XRSNWSA) context deadline exceeded ======LOGS====== Dec 30 05:52:40.362309343 (590h23m40.233392378s) [upload=517565]: Starting upload Dec 30 05:52:40.362391669 (590h23m40.233474704s) [upload=517565]: Done starting upload Dec 30 05:52:40.362497360 (590h23m40.233580405s) [upload=517565]: Uploading segments... Dec 30 05:52:40.362651234 (590h23m40.233734269s) [upload=517565]: Got next segment. Inline: false Dec 30 05:52:40.362695217 (590h23m40.233778262s) [upload=517565]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:52:40.362702251 (590h23m40.233785296s) [upload=517565]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:52:40.605732823 (590h23m40.476815878s) [seg_pos={0 0}, upload=517565]: Begin upload segment... Dec 30 05:52:40.605833094 (590h23m40.476916139s) [seg_pos={0 0}, upload=517565]: Done begin upload segment. Dec 30 05:52:40.605853223 (590h23m40.476936258s) [noise=true , address=77.90.31.13:4626 , piece_id=N7ZYG3AH5IPZUEYHTIIQ63RW7HKIU7YWPXBEXNXVR5J64ZER5SNQ, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.605873130 (590h23m40.476956165s) [noise=true , address=145.79.227.16:4912 , piece_id=UDSOGSEYGTVZ6SXKLDZMNIAHVGHYHZXW5QQSVHYLU65E6FAVGOIQ, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.606603902 (590h23m40.477686947s) [noise=true , address=77.90.31.18:4805 , piece_id=NFGU6L555AAYGED4J6NIZB4OH44CFE65CPKGPSW2W3RIV3SIUGRQ, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.606613411 (590h23m40.477696456s) [noise=true , address=145.79.227.46:4603 , piece_id=PH7UHALDSF5J3XYGDIWVZK5ZT562WC44ZNWM72BCAUIZSLN36INQ, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.607063186 (590h23m40.478146231s) [noise=true , address=145.79.227.32:4834 , piece_id=GPT2MAVR7KYS7V6BYQDAYUOITUYVBDPOUTWYCKLYTTVNVGEOSEAQ, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.607161013 (590h23m40.478244048s) [noise=true , address=145.79.227.2:4113 , piece_id=UQTABKYVKC6OLJY2WPXO64GBXKRLDTZ45FLKKWP6OJGNBS7FA33Q, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.607513103 (590h23m40.478596148s) [noise=true , address=145.79.227.44:4195 , piece_id=2GSVTIU7PC5NLQDW476CKYIK6GQYKCRSWMSA653YRBCMWQAKVL6A, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.607982598 (590h23m40.479065643s) [noise=true , address=145.79.227.10:4401 , piece_id=WVPKVEUNKM6WHKAYBLSYQQILDEGPMFCYGZ2VAVNIHTEQALUVCYFQ, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.608439548 (590h23m40.479522593s) [noise=true , address=45.91.116.14:4190 , piece_id=GKK6HCHFEJSZTZXCMDPRTI533ULZSBRBPKULYEU33E367XRSNWSA, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.609933623 (590h23m40.481016658s) [noise=true , address=145.79.227.42:4122 , piece_id=7USL43IB2DVMZF6R343VZADO7TOHWN6SH5OYD7QNDFK3B6VYJYAQ, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.610524828 (590h23m40.481607874s) [noise=true , address=145.79.227.18:4007 , piece_id=OPI4QXOZVZP2F4KZ6PHQ3WOPFRRBFSRMWO3FRAQNBF5R3DPHOPTQ, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.611078894 (590h23m40.482161939s) [noise=true , address=145.79.227.9:4365 , piece_id=2V6RZZE3IYBV4MKSAJ2R2FPTUNRVQODSEYUCLMRVGHHPB7MIBPXQ, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.613319961 (590h23m40.484403006s) [noise=true , address=45.91.116.12:4227 , piece_id=TMGSOFPHQTXNOB3ZPTXEBJO5PJKFYABG7HLC3MIAYEU6RAAKVWEA, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.614865524 (590h23m40.485948569s) [noise=true , address=45.91.116.13:4153 , piece_id=YNLPZ7DXQYKJZSVJDDDGT2GFIMHNMYHWVDB5KWBDOEZZ7V7AWEDA, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.615092215 (590h23m40.48617527s) [noise=true , address=45.91.116.8:4046 , piece_id=XC47E6Z5T3CHAHMXFXWWK2FL6ADOJXBBAAXHDHQYJIINJE7UGUVA, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.617441990 (590h23m40.488525025s) [noise=true , address=145.79.227.51:4411 , piece_id=EUAMJ3ERXLS5RUIV2VK33SYRTD7KAIQWOGG6CP3KRCUGAIOVWCJQ, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:42.321065425 (590h23m42.19214847s) [upload=517565]: Next returned nil segment Dec 30 05:52:42.321080884 (590h23m42.192163919s) [upload=517565]: Waiting for error group managing segments... Dec 30 05:52:42.596223383 (590h23m42.467306418s) [noise=true , address=145.79.227.16:4912 , piece_id=UDSOGSEYGTVZ6SXKLDZMNIAHVGHYHZXW5QQSVHYLU65E6FAVGOIQ, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:42.607011515 (590h23m42.47809455s) [noise=true , address=145.79.227.51:4411 , piece_id=EUAMJ3ERXLS5RUIV2VK33SYRTD7KAIQWOGG6CP3KRCUGAIOVWCJQ, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:42.610399406 (590h23m42.481482451s) [noise=true , address=145.79.227.42:4122 , piece_id=7USL43IB2DVMZF6R343VZADO7TOHWN6SH5OYD7QNDFK3B6VYJYAQ, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:42.617161733 (590h23m42.488244778s) [noise=true , address=145.79.227.46:4603 , piece_id=PH7UHALDSF5J3XYGDIWVZK5ZT562WC44ZNWM72BCAUIZSLN36INQ, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:42.626384435 (590h23m42.49746747s) [noise=true , address=145.79.227.18:4007 , piece_id=OPI4QXOZVZP2F4KZ6PHQ3WOPFRRBFSRMWO3FRAQNBF5R3DPHOPTQ, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:42.626803593 (590h23m42.497886628s) [noise=true , address=145.79.227.32:4834 , piece_id=GPT2MAVR7KYS7V6BYQDAYUOITUYVBDPOUTWYCKLYTTVNVGEOSEAQ, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:42.627181042 (590h23m42.498264077s) [noise=true , address=145.79.227.2:4113 , piece_id=UQTABKYVKC6OLJY2WPXO64GBXKRLDTZ45FLKKWP6OJGNBS7FA33Q, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:42.635019097 (590h23m42.506102132s) [noise=true , address=145.79.227.9:4365 , piece_id=2V6RZZE3IYBV4MKSAJ2R2FPTUNRVQODSEYUCLMRVGHHPB7MIBPXQ, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:42.678498860 (590h23m42.549581905s) [noise=true , address=145.79.227.10:4401 , piece_id=WVPKVEUNKM6WHKAYBLSYQQILDEGPMFCYGZ2VAVNIHTEQALUVCYFQ, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:42.762451661 (590h23m42.633534696s) [noise=true , address=77.90.31.18:4805 , piece_id=NFGU6L555AAYGED4J6NIZB4OH44CFE65CPKGPSW2W3RIV3SIUGRQ, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:42.921866414 (590h23m42.79294945s) [noise=true , address=45.91.116.13:4153 , piece_id=YNLPZ7DXQYKJZSVJDDDGT2GFIMHNMYHWVDB5KWBDOEZZ7V7AWEDA, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:43.288564190 (590h23m43.159647225s) [noise=true , address=45.91.116.14:4190 , piece_id=GKK6HCHFEJSZTZXCMDPRTI533ULZSBRBPKULYEU33E367XRSNWSA, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:43.687176898 (590h23m43.558259933s) [noise=true , address=45.91.116.8:4046 , piece_id=XC47E6Z5T3CHAHMXFXWWK2FL6ADOJXBBAAXHDHQYJIINJE7UGUVA, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:43.687195162 (590h23m43.558278207s) [seg_pos={0 0}, upload=517565]: Segment reached optimal threshold of 13 pieces. Dec 30 05:52:43.687369003 (590h23m43.558452038s) [noise=true , address=77.90.31.13:4626 , piece_id=N7ZYG3AH5IPZUEYHTIIQ63RW7HKIU7YWPXBEXNXVR5J64ZER5SNQ, seg_pos={0 0}, upload=517565]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 30 05:52:43.687376969 (590h23m43.558460014s) [noise=true , address=77.90.31.13:4626 , piece_id=N7ZYG3AH5IPZUEYHTIIQ63RW7HKIU7YWPXBEXNXVR5J64ZER5SNQ, seg_pos={0 0}, upload=517565]: Long tail cancellation due to optimal threshold Dec 30 05:52:43.687498129 (590h23m43.558581174s) [noise=true , address=145.79.227.44:4195 , piece_id=2GSVTIU7PC5NLQDW476CKYIK6GQYKCRSWMSA653YRBCMWQAKVL6A, seg_pos={0 0}, upload=517565]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 30 05:52:43.687503590 (590h23m43.558586625s) [noise=true , address=145.79.227.44:4195 , piece_id=2GSVTIU7PC5NLQDW476CKYIK6GQYKCRSWMSA653YRBCMWQAKVL6A, seg_pos={0 0}, upload=517565]: Long tail cancellation due to optimal threshold Dec 30 05:52:43.687688292 (590h23m43.558771327s) [noise=true , address=45.91.116.12:4227 , piece_id=TMGSOFPHQTXNOB3ZPTXEBJO5PJKFYABG7HLC3MIAYEU6RAAKVWEA, seg_pos={0 0}, upload=517565]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 30 05:52:43.687704262 (590h23m43.558787297s) [noise=true , address=45.91.116.12:4227 , piece_id=TMGSOFPHQTXNOB3ZPTXEBJO5PJKFYABG7HLC3MIAYEU6RAAKVWEA, seg_pos={0 0}, upload=517565]: Long tail cancellation due to optimal threshold Dec 30 05:52:43.687728829 (590h23m43.558811864s) [seg_pos={0 0}, upload=517565]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:52:43.687815524 (590h23m43.558898569s) [upload=517565]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:52:43.687820954 (590h23m43.558903989s) [upload=517565]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:52:43.983954082 (590h23m43.855037117s) [upload=517565]: Done uploading segments... Dec 30 05:52:43.983989159 (590h23m43.855072194s) [upload=517565]: Upload finished. err: <nil> Duration: 6.493428984s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..X........X................................................] duration: 3.633777192s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:54:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.633802329s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...X........X...............................................] duration: 4.091603913s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:53:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.091631175s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XX...X.XXXXX..X..X...X....X.X.......X..X..X....X.XX.X..X..X.] duration: 6.800852111s trace: https://jaeger.dev.storj.io/trace/18359d6bcf1fdf1f FAIL (2025-12-30 05:53:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: YNLPZ7DXQYKJZSVJDDDGT2GFIMHNMYHWVDB5KWBDOEZZ7V7AWEDA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: XC47E6Z5T3CHAHMXFXWWK2FL6ADOJXBBAAXHDHQYJIINJE7UGUVA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: GKK6HCHFEJSZTZXCMDPRTI533ULZSBRBPKULYEU33E367XRSNWSA) context deadline exceeded ======LOGS====== Dec 30 05:52:40.362309343 (590h23m40.233392378s) [upload=517565]: Starting upload Dec 30 05:52:40.362391669 (590h23m40.233474704s) [upload=517565]: Done starting upload Dec 30 05:52:40.362497360 (590h23m40.233580405s) [upload=517565]: Uploading segments... Dec 30 05:52:40.362651234 (590h23m40.233734269s) [upload=517565]: Got next segment. Inline: false Dec 30 05:52:40.362695217 (590h23m40.233778262s) [upload=517565]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:52:40.362702251 (590h23m40.233785296s) [upload=517565]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:52:40.605732823 (590h23m40.476815878s) [seg_pos={0 0}, upload=517565]: Begin upload segment... Dec 30 05:52:40.605833094 (590h23m40.476916139s) [seg_pos={0 0}, upload=517565]: Done begin upload segment. Dec 30 05:52:40.605853223 (590h23m40.476936258s) [noise=true , address=77.90.31.13:4626 , piece_id=N7ZYG3AH5IPZUEYHTIIQ63RW7HKIU7YWPXBEXNXVR5J64ZER5SNQ, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.605873130 (590h23m40.476956165s) [noise=true , address=145.79.227.16:4912 , piece_id=UDSOGSEYGTVZ6SXKLDZMNIAHVGHYHZXW5QQSVHYLU65E6FAVGOIQ, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.606603902 (590h23m40.477686947s) [noise=true , address=77.90.31.18:4805 , piece_id=NFGU6L555AAYGED4J6NIZB4OH44CFE65CPKGPSW2W3RIV3SIUGRQ, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.606613411 (590h23m40.477696456s) [noise=true , address=145.79.227.46:4603 , piece_id=PH7UHALDSF5J3XYGDIWVZK5ZT562WC44ZNWM72BCAUIZSLN36INQ, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.607063186 (590h23m40.478146231s) [noise=true , address=145.79.227.32:4834 , piece_id=GPT2MAVR7KYS7V6BYQDAYUOITUYVBDPOUTWYCKLYTTVNVGEOSEAQ, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.607161013 (590h23m40.478244048s) [noise=true , address=145.79.227.2:4113 , piece_id=UQTABKYVKC6OLJY2WPXO64GBXKRLDTZ45FLKKWP6OJGNBS7FA33Q, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.607513103 (590h23m40.478596148s) [noise=true , address=145.79.227.44:4195 , piece_id=2GSVTIU7PC5NLQDW476CKYIK6GQYKCRSWMSA653YRBCMWQAKVL6A, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.607982598 (590h23m40.479065643s) [noise=true , address=145.79.227.10:4401 , piece_id=WVPKVEUNKM6WHKAYBLSYQQILDEGPMFCYGZ2VAVNIHTEQALUVCYFQ, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.608439548 (590h23m40.479522593s) [noise=true , address=45.91.116.14:4190 , piece_id=GKK6HCHFEJSZTZXCMDPRTI533ULZSBRBPKULYEU33E367XRSNWSA, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.609933623 (590h23m40.481016658s) [noise=true , address=145.79.227.42:4122 , piece_id=7USL43IB2DVMZF6R343VZADO7TOHWN6SH5OYD7QNDFK3B6VYJYAQ, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.610524828 (590h23m40.481607874s) [noise=true , address=145.79.227.18:4007 , piece_id=OPI4QXOZVZP2F4KZ6PHQ3WOPFRRBFSRMWO3FRAQNBF5R3DPHOPTQ, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.611078894 (590h23m40.482161939s) [noise=true , address=145.79.227.9:4365 , piece_id=2V6RZZE3IYBV4MKSAJ2R2FPTUNRVQODSEYUCLMRVGHHPB7MIBPXQ, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.613319961 (590h23m40.484403006s) [noise=true , address=45.91.116.12:4227 , piece_id=TMGSOFPHQTXNOB3ZPTXEBJO5PJKFYABG7HLC3MIAYEU6RAAKVWEA, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.614865524 (590h23m40.485948569s) [noise=true , address=45.91.116.13:4153 , piece_id=YNLPZ7DXQYKJZSVJDDDGT2GFIMHNMYHWVDB5KWBDOEZZ7V7AWEDA, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.615092215 (590h23m40.48617527s) [noise=true , address=45.91.116.8:4046 , piece_id=XC47E6Z5T3CHAHMXFXWWK2FL6ADOJXBBAAXHDHQYJIINJE7UGUVA, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.617441990 (590h23m40.488525025s) [noise=true , address=145.79.227.51:4411 , piece_id=EUAMJ3ERXLS5RUIV2VK33SYRTD7KAIQWOGG6CP3KRCUGAIOVWCJQ, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:42.321065425 (590h23m42.19214847s) [upload=517565]: Next returned nil segment Dec 30 05:52:42.321080884 (590h23m42.192163919s) [upload=517565]: Waiting for error group managing segments... Dec 30 05:52:42.596223383 (590h23m42.467306418s) [noise=true , address=145.79.227.16:4912 , piece_id=UDSOGSEYGTVZ6SXKLDZMNIAHVGHYHZXW5QQSVHYLU65E6FAVGOIQ, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:42.607011515 (590h23m42.47809455s) [noise=true , address=145.79.227.51:4411 , piece_id=EUAMJ3ERXLS5RUIV2VK33SYRTD7KAIQWOGG6CP3KRCUGAIOVWCJQ, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:42.610399406 (590h23m42.481482451s) [noise=true , address=145.79.227.42:4122 , piece_id=7USL43IB2DVMZF6R343VZADO7TOHWN6SH5OYD7QNDFK3B6VYJYAQ, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:42.617161733 (590h23m42.488244778s) [noise=true , address=145.79.227.46:4603 , piece_id=PH7UHALDSF5J3XYGDIWVZK5ZT562WC44ZNWM72BCAUIZSLN36INQ, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:42.626384435 (590h23m42.49746747s) [noise=true , address=145.79.227.18:4007 , piece_id=OPI4QXOZVZP2F4KZ6PHQ3WOPFRRBFSRMWO3FRAQNBF5R3DPHOPTQ, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:42.626803593 (590h23m42.497886628s) [noise=true , address=145.79.227.32:4834 , piece_id=GPT2MAVR7KYS7V6BYQDAYUOITUYVBDPOUTWYCKLYTTVNVGEOSEAQ, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:42.627181042 (590h23m42.498264077s) [noise=true , address=145.79.227.2:4113 , piece_id=UQTABKYVKC6OLJY2WPXO64GBXKRLDTZ45FLKKWP6OJGNBS7FA33Q, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:42.635019097 (590h23m42.506102132s) [noise=true , address=145.79.227.9:4365 , piece_id=2V6RZZE3IYBV4MKSAJ2R2FPTUNRVQODSEYUCLMRVGHHPB7MIBPXQ, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:42.678498860 (590h23m42.549581905s) [noise=true , address=145.79.227.10:4401 , piece_id=WVPKVEUNKM6WHKAYBLSYQQILDEGPMFCYGZ2VAVNIHTEQALUVCYFQ, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:42.762451661 (590h23m42.633534696s) [noise=true , address=77.90.31.18:4805 , piece_id=NFGU6L555AAYGED4J6NIZB4OH44CFE65CPKGPSW2W3RIV3SIUGRQ, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:42.921866414 (590h23m42.79294945s) [noise=true , address=45.91.116.13:4153 , piece_id=YNLPZ7DXQYKJZSVJDDDGT2GFIMHNMYHWVDB5KWBDOEZZ7V7AWEDA, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:43.288564190 (590h23m43.159647225s) [noise=true , address=45.91.116.14:4190 , piece_id=GKK6HCHFEJSZTZXCMDPRTI533ULZSBRBPKULYEU33E367XRSNWSA, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:43.687176898 (590h23m43.558259933s) [noise=true , address=45.91.116.8:4046 , piece_id=XC47E6Z5T3CHAHMXFXWWK2FL6ADOJXBBAAXHDHQYJIINJE7UGUVA, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:43.687195162 (590h23m43.558278207s) [seg_pos={0 0}, upload=517565]: Segment reached optimal threshold of 13 pieces. Dec 30 05:52:43.687369003 (590h23m43.558452038s) [noise=true , address=77.90.31.13:4626 , piece_id=N7ZYG3AH5IPZUEYHTIIQ63RW7HKIU7YWPXBEXNXVR5J64ZER5SNQ, seg_pos={0 0}, upload=517565]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 30 05:52:43.687376969 (590h23m43.558460014s) [noise=true , address=77.90.31.13:4626 , piece_id=N7ZYG3AH5IPZUEYHTIIQ63RW7HKIU7YWPXBEXNXVR5J64ZER5SNQ, seg_pos={0 0}, upload=517565]: Long tail cancellation due to optimal threshold Dec 30 05:52:43.687498129 (590h23m43.558581174s) [noise=true , address=145.79.227.44:4195 , piece_id=2GSVTIU7PC5NLQDW476CKYIK6GQYKCRSWMSA653YRBCMWQAKVL6A, seg_pos={0 0}, upload=517565]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 30 05:52:43.687503590 (590h23m43.558586625s) [noise=true , address=145.79.227.44:4195 , piece_id=2GSVTIU7PC5NLQDW476CKYIK6GQYKCRSWMSA653YRBCMWQAKVL6A, seg_pos={0 0}, upload=517565]: Long tail cancellation due to optimal threshold Dec 30 05:52:43.687688292 (590h23m43.558771327s) [noise=true , address=45.91.116.12:4227 , piece_id=TMGSOFPHQTXNOB3ZPTXEBJO5PJKFYABG7HLC3MIAYEU6RAAKVWEA, seg_pos={0 0}, upload=517565]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 30 05:52:43.687704262 (590h23m43.558787297s) [noise=true , address=45.91.116.12:4227 , piece_id=TMGSOFPHQTXNOB3ZPTXEBJO5PJKFYABG7HLC3MIAYEU6RAAKVWEA, seg_pos={0 0}, upload=517565]: Long tail cancellation due to optimal threshold Dec 30 05:52:43.687728829 (590h23m43.558811864s) [seg_pos={0 0}, upload=517565]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:52:43.687815524 (590h23m43.558898569s) [upload=517565]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:52:43.687820954 (590h23m43.558903989s) [upload=517565]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:52:43.983954082 (590h23m43.855037117s) [upload=517565]: Done uploading segments... Dec 30 05:52:43.983989159 (590h23m43.855072194s) [upload=517565]: Upload finished. err: <nil> Duration: 6.800885274s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXX...X.XXXXX..X..X...X....X.X.......X..X..X....X.XX.X..X..X] duration: 30.003376271s trace: https://jaeger.dev.storj.io/trace/18359d6bcf1fdf1f FAIL (2025-12-30 05:53:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: YNLPZ7DXQYKJZSVJDDDGT2GFIMHNMYHWVDB5KWBDOEZZ7V7AWEDA) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: XC47E6Z5T3CHAHMXFXWWK2FL6ADOJXBBAAXHDHQYJIINJE7UGUVA) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: GKK6HCHFEJSZTZXCMDPRTI533ULZSBRBPKULYEU33E367XRSNWSA) context deadline exceeded ======LOGS====== Dec 30 05:52:40.362309343 (590h23m40.233392378s) [upload=517565]: Starting upload Dec 30 05:52:40.362391669 (590h23m40.233474704s) [upload=517565]: Done starting upload Dec 30 05:52:40.362497360 (590h23m40.233580405s) [upload=517565]: Uploading segments... Dec 30 05:52:40.362651234 (590h23m40.233734269s) [upload=517565]: Got next segment. Inline: false Dec 30 05:52:40.362695217 (590h23m40.233778262s) [upload=517565]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:52:40.362702251 (590h23m40.233785296s) [upload=517565]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:52:40.605732823 (590h23m40.476815878s) [seg_pos={0 0}, upload=517565]: Begin upload segment... Dec 30 05:52:40.605833094 (590h23m40.476916139s) [seg_pos={0 0}, upload=517565]: Done begin upload segment. Dec 30 05:52:40.605853223 (590h23m40.476936258s) [noise=true , address=77.90.31.13:4626 , piece_id=N7ZYG3AH5IPZUEYHTIIQ63RW7HKIU7YWPXBEXNXVR5J64ZER5SNQ, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.605873130 (590h23m40.476956165s) [noise=true , address=145.79.227.16:4912 , piece_id=UDSOGSEYGTVZ6SXKLDZMNIAHVGHYHZXW5QQSVHYLU65E6FAVGOIQ, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.606603902 (590h23m40.477686947s) [noise=true , address=77.90.31.18:4805 , piece_id=NFGU6L555AAYGED4J6NIZB4OH44CFE65CPKGPSW2W3RIV3SIUGRQ, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.606613411 (590h23m40.477696456s) [noise=true , address=145.79.227.46:4603 , piece_id=PH7UHALDSF5J3XYGDIWVZK5ZT562WC44ZNWM72BCAUIZSLN36INQ, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.607063186 (590h23m40.478146231s) [noise=true , address=145.79.227.32:4834 , piece_id=GPT2MAVR7KYS7V6BYQDAYUOITUYVBDPOUTWYCKLYTTVNVGEOSEAQ, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.607161013 (590h23m40.478244048s) [noise=true , address=145.79.227.2:4113 , piece_id=UQTABKYVKC6OLJY2WPXO64GBXKRLDTZ45FLKKWP6OJGNBS7FA33Q, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.607513103 (590h23m40.478596148s) [noise=true , address=145.79.227.44:4195 , piece_id=2GSVTIU7PC5NLQDW476CKYIK6GQYKCRSWMSA653YRBCMWQAKVL6A, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.607982598 (590h23m40.479065643s) [noise=true , address=145.79.227.10:4401 , piece_id=WVPKVEUNKM6WHKAYBLSYQQILDEGPMFCYGZ2VAVNIHTEQALUVCYFQ, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.608439548 (590h23m40.479522593s) [noise=true , address=45.91.116.14:4190 , piece_id=GKK6HCHFEJSZTZXCMDPRTI533ULZSBRBPKULYEU33E367XRSNWSA, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.609933623 (590h23m40.481016658s) [noise=true , address=145.79.227.42:4122 , piece_id=7USL43IB2DVMZF6R343VZADO7TOHWN6SH5OYD7QNDFK3B6VYJYAQ, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.610524828 (590h23m40.481607874s) [noise=true , address=145.79.227.18:4007 , piece_id=OPI4QXOZVZP2F4KZ6PHQ3WOPFRRBFSRMWO3FRAQNBF5R3DPHOPTQ, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.611078894 (590h23m40.482161939s) [noise=true , address=145.79.227.9:4365 , piece_id=2V6RZZE3IYBV4MKSAJ2R2FPTUNRVQODSEYUCLMRVGHHPB7MIBPXQ, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.613319961 (590h23m40.484403006s) [noise=true , address=45.91.116.12:4227 , piece_id=TMGSOFPHQTXNOB3ZPTXEBJO5PJKFYABG7HLC3MIAYEU6RAAKVWEA, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.614865524 (590h23m40.485948569s) [noise=true , address=45.91.116.13:4153 , piece_id=YNLPZ7DXQYKJZSVJDDDGT2GFIMHNMYHWVDB5KWBDOEZZ7V7AWEDA, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.615092215 (590h23m40.48617527s) [noise=true , address=45.91.116.8:4046 , piece_id=XC47E6Z5T3CHAHMXFXWWK2FL6ADOJXBBAAXHDHQYJIINJE7UGUVA, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:40.617441990 (590h23m40.488525025s) [noise=true , address=145.79.227.51:4411 , piece_id=EUAMJ3ERXLS5RUIV2VK33SYRTD7KAIQWOGG6CP3KRCUGAIOVWCJQ, seg_pos={0 0}, upload=517565]: Uploading piece... Dec 30 05:52:42.321065425 (590h23m42.19214847s) [upload=517565]: Next returned nil segment Dec 30 05:52:42.321080884 (590h23m42.192163919s) [upload=517565]: Waiting for error group managing segments... Dec 30 05:52:42.596223383 (590h23m42.467306418s) [noise=true , address=145.79.227.16:4912 , piece_id=UDSOGSEYGTVZ6SXKLDZMNIAHVGHYHZXW5QQSVHYLU65E6FAVGOIQ, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:42.607011515 (590h23m42.47809455s) [noise=true , address=145.79.227.51:4411 , piece_id=EUAMJ3ERXLS5RUIV2VK33SYRTD7KAIQWOGG6CP3KRCUGAIOVWCJQ, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:42.610399406 (590h23m42.481482451s) [noise=true , address=145.79.227.42:4122 , piece_id=7USL43IB2DVMZF6R343VZADO7TOHWN6SH5OYD7QNDFK3B6VYJYAQ, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:42.617161733 (590h23m42.488244778s) [noise=true , address=145.79.227.46:4603 , piece_id=PH7UHALDSF5J3XYGDIWVZK5ZT562WC44ZNWM72BCAUIZSLN36INQ, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:42.626384435 (590h23m42.49746747s) [noise=true , address=145.79.227.18:4007 , piece_id=OPI4QXOZVZP2F4KZ6PHQ3WOPFRRBFSRMWO3FRAQNBF5R3DPHOPTQ, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:42.626803593 (590h23m42.497886628s) [noise=true , address=145.79.227.32:4834 , piece_id=GPT2MAVR7KYS7V6BYQDAYUOITUYVBDPOUTWYCKLYTTVNVGEOSEAQ, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:42.627181042 (590h23m42.498264077s) [noise=true , address=145.79.227.2:4113 , piece_id=UQTABKYVKC6OLJY2WPXO64GBXKRLDTZ45FLKKWP6OJGNBS7FA33Q, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:42.635019097 (590h23m42.506102132s) [noise=true , address=145.79.227.9:4365 , piece_id=2V6RZZE3IYBV4MKSAJ2R2FPTUNRVQODSEYUCLMRVGHHPB7MIBPXQ, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:42.678498860 (590h23m42.549581905s) [noise=true , address=145.79.227.10:4401 , piece_id=WVPKVEUNKM6WHKAYBLSYQQILDEGPMFCYGZ2VAVNIHTEQALUVCYFQ, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:42.762451661 (590h23m42.633534696s) [noise=true , address=77.90.31.18:4805 , piece_id=NFGU6L555AAYGED4J6NIZB4OH44CFE65CPKGPSW2W3RIV3SIUGRQ, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:42.921866414 (590h23m42.79294945s) [noise=true , address=45.91.116.13:4153 , piece_id=YNLPZ7DXQYKJZSVJDDDGT2GFIMHNMYHWVDB5KWBDOEZZ7V7AWEDA, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:43.288564190 (590h23m43.159647225s) [noise=true , address=45.91.116.14:4190 , piece_id=GKK6HCHFEJSZTZXCMDPRTI533ULZSBRBPKULYEU33E367XRSNWSA, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:43.687176898 (590h23m43.558259933s) [noise=true , address=45.91.116.8:4046 , piece_id=XC47E6Z5T3CHAHMXFXWWK2FL6ADOJXBBAAXHDHQYJIINJE7UGUVA, seg_pos={0 0}, upload=517565]: Done uploading piece. err: <nil> Dec 30 05:52:43.687195162 (590h23m43.558278207s) [seg_pos={0 0}, upload=517565]: Segment reached optimal threshold of 13 pieces. Dec 30 05:52:43.687369003 (590h23m43.558452038s) [noise=true , address=77.90.31.13:4626 , piece_id=N7ZYG3AH5IPZUEYHTIIQ63RW7HKIU7YWPXBEXNXVR5J64ZER5SNQ, seg_pos={0 0}, upload=517565]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Dec 30 05:52:43.687376969 (590h23m43.558460014s) [noise=true , address=77.90.31.13:4626 , piece_id=N7ZYG3AH5IPZUEYHTIIQ63RW7HKIU7YWPXBEXNXVR5J64ZER5SNQ, seg_pos={0 0}, upload=517565]: Long tail cancellation due to optimal threshold Dec 30 05:52:43.687498129 (590h23m43.558581174s) [noise=true , address=145.79.227.44:4195 , piece_id=2GSVTIU7PC5NLQDW476CKYIK6GQYKCRSWMSA653YRBCMWQAKVL6A, seg_pos={0 0}, upload=517565]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF Dec 30 05:52:43.687503590 (590h23m43.558586625s) [noise=true , address=145.79.227.44:4195 , piece_id=2GSVTIU7PC5NLQDW476CKYIK6GQYKCRSWMSA653YRBCMWQAKVL6A, seg_pos={0 0}, upload=517565]: Long tail cancellation due to optimal threshold Dec 30 05:52:43.687688292 (590h23m43.558771327s) [noise=true , address=45.91.116.12:4227 , piece_id=TMGSOFPHQTXNOB3ZPTXEBJO5PJKFYABG7HLC3MIAYEU6RAAKVWEA, seg_pos={0 0}, upload=517565]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 30 05:52:43.687704262 (590h23m43.558787297s) [noise=true , address=45.91.116.12:4227 , piece_id=TMGSOFPHQTXNOB3ZPTXEBJO5PJKFYABG7HLC3MIAYEU6RAAKVWEA, seg_pos={0 0}, upload=517565]: Long tail cancellation due to optimal threshold Dec 30 05:52:43.687728829 (590h23m43.558811864s) [seg_pos={0 0}, upload=517565]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:52:43.687815524 (590h23m43.558898569s) [upload=517565]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:52:43.687820954 (590h23m43.558903989s) [upload=517565]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:52:43.983954082 (590h23m43.855037117s) [upload=517565]: Done uploading segments... Dec 30 05:52:43.983989159 (590h23m43.855072194s) [upload=517565]: Upload finished. err: <nil> Duration: 30.003403794s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....X........X..............................................] duration: 5.000969574s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:52:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000994692s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [X....X........X.............................................] duration: 4.275883644s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:51:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.27590275s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XXX...X.XXXXX..X..X...X....X.X.......X..X..X....X.XX.X..X..] duration: 5.971674963s trace: https://jaeger.dev.storj.io/trace/458d34df16dda41f FAIL (2025-12-30 05:51:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 05:50:07.775176611 (590h21m7.646259657s) [upload=517528]: Starting upload Dec 30 05:50:07.775283846 (590h21m7.646366892s) [upload=517528]: Done starting upload Dec 30 05:50:07.775550834 (590h21m7.646633879s) [upload=517528]: Uploading segments... Dec 30 05:50:07.775630757 (590h21m7.646713822s) [upload=517528]: Got next segment. Inline: false Dec 30 05:50:07.775680331 (590h21m7.646763386s) [upload=517528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:50:07.775686753 (590h21m7.646769798s) [upload=517528]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:50:08.019770230 (590h21m7.890853276s) [seg_pos={0 0}, upload=517528]: Begin upload segment... Dec 30 05:50:08.019832850 (590h21m7.890915895s) [seg_pos={0 0}, upload=517528]: Done begin upload segment. Dec 30 05:50:08.019869389 (590h21m7.890952435s) [noise=true , address=45.91.116.11:4120 , piece_id=ZQXA4J2LZLFAJM2CPJNID267RGWBVN6D4OAZ44OXM43WKTBFV2UA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.019912181 (590h21m7.890995226s) [noise=true , address=45.91.116.12:4226 , piece_id=2VHV7LSCJQA3L4TCDEODEAQL3J2MFJ2U23D6UML7N7TQRUSYVB4Q, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.020546919 (590h21m7.891630014s) [noise=true , address=77.90.31.15:4700 , piece_id=OMFZWUMEDLVNDKATUIXFLZUKCDR3PQRI7AZ4JFK75QQ7N22SWEVQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.021090925 (590h21m7.89217397s) [noise=true , address=77.90.31.13:4627 , piece_id=T3S7QE7W2SJV7IOLF3SH3PZA3ARIGT5FZPDYC7GN3MKM6XTB5GPA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.021775438 (590h21m7.892858473s) [noise=true , address=45.91.116.13:4153 , piece_id=HXDPZTZ4SPH2MJBKC5PLGKZDOXKJOQZI3LGMU3BV6MGYIMKTEI7Q, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.022911031 (590h21m7.893994076s) [noise=true , address=145.79.227.23:4545 , piece_id=MZXZHBGJEW6HQM7V2C55QGQIAEBI5ONN3MWWYMTHHPQBRJRQZNPQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.023521945 (590h21m7.89460499s) [noise=true , address=145.79.227.9:4367 , piece_id=QHJ337HTW2NO3MXCITYPPE443FKHLPL5IG5N6BXIBPSJJWV6LTOQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.025375694 (590h21m7.896458739s) [noise=true , address=145.79.227.15:4875 , piece_id=PBOBRWD3KAFZJYTWVND3Y36RO5XY7RVIYDCR6RV55NWPLBGGZCBQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.025510461 (590h21m7.896593496s) [noise=true , address=145.79.227.48:4302 , piece_id=H6HID65U74FXC7VS5E52IPRC35DLKFCKZOUO3QGDU5CXHLMVVBPA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.027279870 (590h21m7.898362905s) [noise=true , address=145.79.227.16:4910 , piece_id=A4I62LSDXLILZECT5CPFSDBZRZLXY4IHZKDJUETACHGXXJJNCUNA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.028414881 (590h21m7.899497926s) [noise=true , address=45.91.116.7:4010 , piece_id=RMXZNAEVMZZ2KPBZKSYQLEZAKAFRKH4LJ55TQLT6XXXSUVRRQBIQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.029089216 (590h21m7.900172261s) [noise=true , address=145.79.227.5:4223 , piece_id=X6CJNG227YK7H3FGKWKBOKSSPH44TZ5FEV2EHVHPALB7Y7CNP3AA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.030922245 (590h21m7.902005291s) [noise=true , address=77.90.31.16:4735 , piece_id=WK6DWLTVPXCGINS5QBSCG6CHYXUDMM36ICUPG7TMF2F7XXTX6CIA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.032664954 (590h21m7.903747989s) [noise=true , address=77.90.31.11:4555 , piece_id=U2GJ7KI4S6RHCDJWNOEJXAU4NV7V3ODOF3BGNICVIZLBDHF3H7JQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.034793537 (590h21m7.905876582s) [noise=true , address=45.91.116.4:4940 , piece_id=SHHV2UXBGHGDSY5K46NIWCMN7FA3OW66C5BVKNTLJI2TGTUAVQCQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.036058316 (590h21m7.907141361s) [noise=true , address=45.91.116.8:4048 , piece_id=HPQ2ZE57MQ2ZRUQZ3B5SWUPENFTEAXTM3U3ECRWBGEQLWD2BBHKQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:09.727263063 (590h21m9.598346108s) [upload=517528]: Next returned nil segment Dec 30 05:50:09.727283351 (590h21m9.598366386s) [upload=517528]: Waiting for error group managing segments... Dec 30 05:50:10.010280438 (590h21m9.881363483s) [noise=true , address=145.79.227.5:4223 , piece_id=X6CJNG227YK7H3FGKWKBOKSSPH44TZ5FEV2EHVHPALB7Y7CNP3AA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.015210825 (590h21m9.88629387s) [noise=true , address=145.79.227.48:4302 , piece_id=H6HID65U74FXC7VS5E52IPRC35DLKFCKZOUO3QGDU5CXHLMVVBPA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.018545656 (590h21m9.889628701s) [noise=true , address=145.79.227.9:4367 , piece_id=QHJ337HTW2NO3MXCITYPPE443FKHLPL5IG5N6BXIBPSJJWV6LTOQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.043972091 (590h21m9.915055137s) [noise=true , address=145.79.227.16:4910 , piece_id=A4I62LSDXLILZECT5CPFSDBZRZLXY4IHZKDJUETACHGXXJJNCUNA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.047623925 (590h21m9.91870697s) [noise=true , address=145.79.227.23:4545 , piece_id=MZXZHBGJEW6HQM7V2C55QGQIAEBI5ONN3MWWYMTHHPQBRJRQZNPQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.102492926 (590h21m9.973575971s) [noise=true , address=145.79.227.15:4875 , piece_id=PBOBRWD3KAFZJYTWVND3Y36RO5XY7RVIYDCR6RV55NWPLBGGZCBQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.214049859 (590h21m10.085132894s) [noise=true , address=77.90.31.16:4735 , piece_id=WK6DWLTVPXCGINS5QBSCG6CHYXUDMM36ICUPG7TMF2F7XXTX6CIA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.311122282 (590h21m10.182205327s) [noise=true , address=77.90.31.15:4700 , piece_id=OMFZWUMEDLVNDKATUIXFLZUKCDR3PQRI7AZ4JFK75QQ7N22SWEVQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.327549262 (590h21m10.198632297s) [noise=true , address=77.90.31.11:4555 , piece_id=U2GJ7KI4S6RHCDJWNOEJXAU4NV7V3ODOF3BGNICVIZLBDHF3H7JQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.356830358 (590h21m10.227913403s) [noise=true , address=45.91.116.12:4226 , piece_id=2VHV7LSCJQA3L4TCDEODEAQL3J2MFJ2U23D6UML7N7TQRUSYVB4Q, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.419134055 (590h21m10.290217101s) [noise=true , address=45.91.116.4:4940 , piece_id=SHHV2UXBGHGDSY5K46NIWCMN7FA3OW66C5BVKNTLJI2TGTUAVQCQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.481996056 (590h21m10.353079101s) [noise=true , address=45.91.116.11:4120 , piece_id=ZQXA4J2LZLFAJM2CPJNID267RGWBVN6D4OAZ44OXM43WKTBFV2UA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:11.021539821 (590h21m10.892622856s) [noise=true , address=77.90.31.13:4627 , piece_id=T3S7QE7W2SJV7IOLF3SH3PZA3ARIGT5FZPDYC7GN3MKM6XTB5GPA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:11.021569967 (590h21m10.892653012s) [seg_pos={0 0}, upload=517528]: Segment reached optimal threshold of 13 pieces. Dec 30 05:50:11.021789777 (590h21m10.892872822s) [noise=true , address=45.91.116.7:4010 , piece_id=RMXZNAEVMZZ2KPBZKSYQLEZAKAFRKH4LJ55TQLT6XXXSUVRRQBIQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 30 05:50:11.021826537 (590h21m10.892909582s) [noise=true , address=45.91.116.7:4010 , piece_id=RMXZNAEVMZZ2KPBZKSYQLEZAKAFRKH4LJ55TQLT6XXXSUVRRQBIQ, seg_pos={0 0}, upload=517528]: Long tail cancellation due to optimal threshold Dec 30 05:50:11.021899866 (590h21m10.892982911s) [noise=true , address=45.91.116.8:4048 , piece_id=HPQ2ZE57MQ2ZRUQZ3B5SWUPENFTEAXTM3U3ECRWBGEQLWD2BBHKQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 30 05:50:11.021908542 (590h21m10.892991577s) [noise=true , address=45.91.116.8:4048 , piece_id=HPQ2ZE57MQ2ZRUQZ3B5SWUPENFTEAXTM3U3ECRWBGEQLWD2BBHKQ, seg_pos={0 0}, upload=517528]: Long tail cancellation due to optimal threshold Dec 30 05:50:11.022066914 (590h21m10.893149949s) [noise=true , address=45.91.116.13:4153 , piece_id=HXDPZTZ4SPH2MJBKC5PLGKZDOXKJOQZI3LGMU3BV6MGYIMKTEI7Q, seg_pos={0 0}, upload=517528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 30 05:50:11.022074408 (590h21m10.893157443s) [noise=true , address=45.91.116.13:4153 , piece_id=HXDPZTZ4SPH2MJBKC5PLGKZDOXKJOQZI3LGMU3BV6MGYIMKTEI7Q, seg_pos={0 0}, upload=517528]: Long tail cancellation due to optimal threshold Dec 30 05:50:11.022094626 (590h21m10.893177671s) [seg_pos={0 0}, upload=517528]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:50:11.022160422 (590h21m10.893243457s) [upload=517528]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:50:11.022165092 (590h21m10.893248127s) [upload=517528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:50:11.313202827 (590h21m11.184285872s) [upload=517528]: Done uploading segments... Dec 30 05:50:11.313228124 (590h21m11.18431117s) [upload=517528]: Upload finished. err: <nil> Duration: 5.971700924s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.XXX...X.XXXXX..X..X...X....X.X.......X..X..X....X.XX.X..X.] duration: 6.046296485s trace: https://jaeger.dev.storj.io/trace/458d34df16dda41f FAIL (2025-12-30 05:51:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 05:50:07.775176611 (590h21m7.646259657s) [upload=517528]: Starting upload Dec 30 05:50:07.775283846 (590h21m7.646366892s) [upload=517528]: Done starting upload Dec 30 05:50:07.775550834 (590h21m7.646633879s) [upload=517528]: Uploading segments... Dec 30 05:50:07.775630757 (590h21m7.646713822s) [upload=517528]: Got next segment. Inline: false Dec 30 05:50:07.775680331 (590h21m7.646763386s) [upload=517528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:50:07.775686753 (590h21m7.646769798s) [upload=517528]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:50:08.019770230 (590h21m7.890853276s) [seg_pos={0 0}, upload=517528]: Begin upload segment... Dec 30 05:50:08.019832850 (590h21m7.890915895s) [seg_pos={0 0}, upload=517528]: Done begin upload segment. Dec 30 05:50:08.019869389 (590h21m7.890952435s) [noise=true , address=45.91.116.11:4120 , piece_id=ZQXA4J2LZLFAJM2CPJNID267RGWBVN6D4OAZ44OXM43WKTBFV2UA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.019912181 (590h21m7.890995226s) [noise=true , address=45.91.116.12:4226 , piece_id=2VHV7LSCJQA3L4TCDEODEAQL3J2MFJ2U23D6UML7N7TQRUSYVB4Q, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.020546919 (590h21m7.891630014s) [noise=true , address=77.90.31.15:4700 , piece_id=OMFZWUMEDLVNDKATUIXFLZUKCDR3PQRI7AZ4JFK75QQ7N22SWEVQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.021090925 (590h21m7.89217397s) [noise=true , address=77.90.31.13:4627 , piece_id=T3S7QE7W2SJV7IOLF3SH3PZA3ARIGT5FZPDYC7GN3MKM6XTB5GPA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.021775438 (590h21m7.892858473s) [noise=true , address=45.91.116.13:4153 , piece_id=HXDPZTZ4SPH2MJBKC5PLGKZDOXKJOQZI3LGMU3BV6MGYIMKTEI7Q, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.022911031 (590h21m7.893994076s) [noise=true , address=145.79.227.23:4545 , piece_id=MZXZHBGJEW6HQM7V2C55QGQIAEBI5ONN3MWWYMTHHPQBRJRQZNPQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.023521945 (590h21m7.89460499s) [noise=true , address=145.79.227.9:4367 , piece_id=QHJ337HTW2NO3MXCITYPPE443FKHLPL5IG5N6BXIBPSJJWV6LTOQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.025375694 (590h21m7.896458739s) [noise=true , address=145.79.227.15:4875 , piece_id=PBOBRWD3KAFZJYTWVND3Y36RO5XY7RVIYDCR6RV55NWPLBGGZCBQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.025510461 (590h21m7.896593496s) [noise=true , address=145.79.227.48:4302 , piece_id=H6HID65U74FXC7VS5E52IPRC35DLKFCKZOUO3QGDU5CXHLMVVBPA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.027279870 (590h21m7.898362905s) [noise=true , address=145.79.227.16:4910 , piece_id=A4I62LSDXLILZECT5CPFSDBZRZLXY4IHZKDJUETACHGXXJJNCUNA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.028414881 (590h21m7.899497926s) [noise=true , address=45.91.116.7:4010 , piece_id=RMXZNAEVMZZ2KPBZKSYQLEZAKAFRKH4LJ55TQLT6XXXSUVRRQBIQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.029089216 (590h21m7.900172261s) [noise=true , address=145.79.227.5:4223 , piece_id=X6CJNG227YK7H3FGKWKBOKSSPH44TZ5FEV2EHVHPALB7Y7CNP3AA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.030922245 (590h21m7.902005291s) [noise=true , address=77.90.31.16:4735 , piece_id=WK6DWLTVPXCGINS5QBSCG6CHYXUDMM36ICUPG7TMF2F7XXTX6CIA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.032664954 (590h21m7.903747989s) [noise=true , address=77.90.31.11:4555 , piece_id=U2GJ7KI4S6RHCDJWNOEJXAU4NV7V3ODOF3BGNICVIZLBDHF3H7JQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.034793537 (590h21m7.905876582s) [noise=true , address=45.91.116.4:4940 , piece_id=SHHV2UXBGHGDSY5K46NIWCMN7FA3OW66C5BVKNTLJI2TGTUAVQCQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.036058316 (590h21m7.907141361s) [noise=true , address=45.91.116.8:4048 , piece_id=HPQ2ZE57MQ2ZRUQZ3B5SWUPENFTEAXTM3U3ECRWBGEQLWD2BBHKQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:09.727263063 (590h21m9.598346108s) [upload=517528]: Next returned nil segment Dec 30 05:50:09.727283351 (590h21m9.598366386s) [upload=517528]: Waiting for error group managing segments... Dec 30 05:50:10.010280438 (590h21m9.881363483s) [noise=true , address=145.79.227.5:4223 , piece_id=X6CJNG227YK7H3FGKWKBOKSSPH44TZ5FEV2EHVHPALB7Y7CNP3AA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.015210825 (590h21m9.88629387s) [noise=true , address=145.79.227.48:4302 , piece_id=H6HID65U74FXC7VS5E52IPRC35DLKFCKZOUO3QGDU5CXHLMVVBPA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.018545656 (590h21m9.889628701s) [noise=true , address=145.79.227.9:4367 , piece_id=QHJ337HTW2NO3MXCITYPPE443FKHLPL5IG5N6BXIBPSJJWV6LTOQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.043972091 (590h21m9.915055137s) [noise=true , address=145.79.227.16:4910 , piece_id=A4I62LSDXLILZECT5CPFSDBZRZLXY4IHZKDJUETACHGXXJJNCUNA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.047623925 (590h21m9.91870697s) [noise=true , address=145.79.227.23:4545 , piece_id=MZXZHBGJEW6HQM7V2C55QGQIAEBI5ONN3MWWYMTHHPQBRJRQZNPQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.102492926 (590h21m9.973575971s) [noise=true , address=145.79.227.15:4875 , piece_id=PBOBRWD3KAFZJYTWVND3Y36RO5XY7RVIYDCR6RV55NWPLBGGZCBQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.214049859 (590h21m10.085132894s) [noise=true , address=77.90.31.16:4735 , piece_id=WK6DWLTVPXCGINS5QBSCG6CHYXUDMM36ICUPG7TMF2F7XXTX6CIA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.311122282 (590h21m10.182205327s) [noise=true , address=77.90.31.15:4700 , piece_id=OMFZWUMEDLVNDKATUIXFLZUKCDR3PQRI7AZ4JFK75QQ7N22SWEVQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.327549262 (590h21m10.198632297s) [noise=true , address=77.90.31.11:4555 , piece_id=U2GJ7KI4S6RHCDJWNOEJXAU4NV7V3ODOF3BGNICVIZLBDHF3H7JQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.356830358 (590h21m10.227913403s) [noise=true , address=45.91.116.12:4226 , piece_id=2VHV7LSCJQA3L4TCDEODEAQL3J2MFJ2U23D6UML7N7TQRUSYVB4Q, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.419134055 (590h21m10.290217101s) [noise=true , address=45.91.116.4:4940 , piece_id=SHHV2UXBGHGDSY5K46NIWCMN7FA3OW66C5BVKNTLJI2TGTUAVQCQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.481996056 (590h21m10.353079101s) [noise=true , address=45.91.116.11:4120 , piece_id=ZQXA4J2LZLFAJM2CPJNID267RGWBVN6D4OAZ44OXM43WKTBFV2UA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:11.021539821 (590h21m10.892622856s) [noise=true , address=77.90.31.13:4627 , piece_id=T3S7QE7W2SJV7IOLF3SH3PZA3ARIGT5FZPDYC7GN3MKM6XTB5GPA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:11.021569967 (590h21m10.892653012s) [seg_pos={0 0}, upload=517528]: Segment reached optimal threshold of 13 pieces. Dec 30 05:50:11.021789777 (590h21m10.892872822s) [noise=true , address=45.91.116.7:4010 , piece_id=RMXZNAEVMZZ2KPBZKSYQLEZAKAFRKH4LJ55TQLT6XXXSUVRRQBIQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 30 05:50:11.021826537 (590h21m10.892909582s) [noise=true , address=45.91.116.7:4010 , piece_id=RMXZNAEVMZZ2KPBZKSYQLEZAKAFRKH4LJ55TQLT6XXXSUVRRQBIQ, seg_pos={0 0}, upload=517528]: Long tail cancellation due to optimal threshold Dec 30 05:50:11.021899866 (590h21m10.892982911s) [noise=true , address=45.91.116.8:4048 , piece_id=HPQ2ZE57MQ2ZRUQZ3B5SWUPENFTEAXTM3U3ECRWBGEQLWD2BBHKQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 30 05:50:11.021908542 (590h21m10.892991577s) [noise=true , address=45.91.116.8:4048 , piece_id=HPQ2ZE57MQ2ZRUQZ3B5SWUPENFTEAXTM3U3ECRWBGEQLWD2BBHKQ, seg_pos={0 0}, upload=517528]: Long tail cancellation due to optimal threshold Dec 30 05:50:11.022066914 (590h21m10.893149949s) [noise=true , address=45.91.116.13:4153 , piece_id=HXDPZTZ4SPH2MJBKC5PLGKZDOXKJOQZI3LGMU3BV6MGYIMKTEI7Q, seg_pos={0 0}, upload=517528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 30 05:50:11.022074408 (590h21m10.893157443s) [noise=true , address=45.91.116.13:4153 , piece_id=HXDPZTZ4SPH2MJBKC5PLGKZDOXKJOQZI3LGMU3BV6MGYIMKTEI7Q, seg_pos={0 0}, upload=517528]: Long tail cancellation due to optimal threshold Dec 30 05:50:11.022094626 (590h21m10.893177671s) [seg_pos={0 0}, upload=517528]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:50:11.022160422 (590h21m10.893243457s) [upload=517528]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:50:11.022165092 (590h21m10.893248127s) [upload=517528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:50:11.313202827 (590h21m11.184285872s) [upload=517528]: Done uploading segments... Dec 30 05:50:11.313228124 (590h21m11.18431117s) [upload=517528]: Upload finished. err: <nil> Duration: 6.046323547s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.X....X........X............................................] duration: 3.488787055s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:50:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.488816131s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XX.XXX...X.XXXXX..X..X...X....X.X.......X..X..X....X.XX.X..X] duration: 30.003444139s trace: https://jaeger.dev.storj.io/trace/458d34df16dda41f FAIL (2025-12-30 05:50:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 05:50:07.775176611 (590h21m7.646259657s) [upload=517528]: Starting upload Dec 30 05:50:07.775283846 (590h21m7.646366892s) [upload=517528]: Done starting upload Dec 30 05:50:07.775550834 (590h21m7.646633879s) [upload=517528]: Uploading segments... Dec 30 05:50:07.775630757 (590h21m7.646713822s) [upload=517528]: Got next segment. Inline: false Dec 30 05:50:07.775680331 (590h21m7.646763386s) [upload=517528]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:50:07.775686753 (590h21m7.646769798s) [upload=517528]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:50:08.019770230 (590h21m7.890853276s) [seg_pos={0 0}, upload=517528]: Begin upload segment... Dec 30 05:50:08.019832850 (590h21m7.890915895s) [seg_pos={0 0}, upload=517528]: Done begin upload segment. Dec 30 05:50:08.019869389 (590h21m7.890952435s) [noise=true , address=45.91.116.11:4120 , piece_id=ZQXA4J2LZLFAJM2CPJNID267RGWBVN6D4OAZ44OXM43WKTBFV2UA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.019912181 (590h21m7.890995226s) [noise=true , address=45.91.116.12:4226 , piece_id=2VHV7LSCJQA3L4TCDEODEAQL3J2MFJ2U23D6UML7N7TQRUSYVB4Q, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.020546919 (590h21m7.891630014s) [noise=true , address=77.90.31.15:4700 , piece_id=OMFZWUMEDLVNDKATUIXFLZUKCDR3PQRI7AZ4JFK75QQ7N22SWEVQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.021090925 (590h21m7.89217397s) [noise=true , address=77.90.31.13:4627 , piece_id=T3S7QE7W2SJV7IOLF3SH3PZA3ARIGT5FZPDYC7GN3MKM6XTB5GPA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.021775438 (590h21m7.892858473s) [noise=true , address=45.91.116.13:4153 , piece_id=HXDPZTZ4SPH2MJBKC5PLGKZDOXKJOQZI3LGMU3BV6MGYIMKTEI7Q, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.022911031 (590h21m7.893994076s) [noise=true , address=145.79.227.23:4545 , piece_id=MZXZHBGJEW6HQM7V2C55QGQIAEBI5ONN3MWWYMTHHPQBRJRQZNPQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.023521945 (590h21m7.89460499s) [noise=true , address=145.79.227.9:4367 , piece_id=QHJ337HTW2NO3MXCITYPPE443FKHLPL5IG5N6BXIBPSJJWV6LTOQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.025375694 (590h21m7.896458739s) [noise=true , address=145.79.227.15:4875 , piece_id=PBOBRWD3KAFZJYTWVND3Y36RO5XY7RVIYDCR6RV55NWPLBGGZCBQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.025510461 (590h21m7.896593496s) [noise=true , address=145.79.227.48:4302 , piece_id=H6HID65U74FXC7VS5E52IPRC35DLKFCKZOUO3QGDU5CXHLMVVBPA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.027279870 (590h21m7.898362905s) [noise=true , address=145.79.227.16:4910 , piece_id=A4I62LSDXLILZECT5CPFSDBZRZLXY4IHZKDJUETACHGXXJJNCUNA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.028414881 (590h21m7.899497926s) [noise=true , address=45.91.116.7:4010 , piece_id=RMXZNAEVMZZ2KPBZKSYQLEZAKAFRKH4LJ55TQLT6XXXSUVRRQBIQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.029089216 (590h21m7.900172261s) [noise=true , address=145.79.227.5:4223 , piece_id=X6CJNG227YK7H3FGKWKBOKSSPH44TZ5FEV2EHVHPALB7Y7CNP3AA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.030922245 (590h21m7.902005291s) [noise=true , address=77.90.31.16:4735 , piece_id=WK6DWLTVPXCGINS5QBSCG6CHYXUDMM36ICUPG7TMF2F7XXTX6CIA, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.032664954 (590h21m7.903747989s) [noise=true , address=77.90.31.11:4555 , piece_id=U2GJ7KI4S6RHCDJWNOEJXAU4NV7V3ODOF3BGNICVIZLBDHF3H7JQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.034793537 (590h21m7.905876582s) [noise=true , address=45.91.116.4:4940 , piece_id=SHHV2UXBGHGDSY5K46NIWCMN7FA3OW66C5BVKNTLJI2TGTUAVQCQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:08.036058316 (590h21m7.907141361s) [noise=true , address=45.91.116.8:4048 , piece_id=HPQ2ZE57MQ2ZRUQZ3B5SWUPENFTEAXTM3U3ECRWBGEQLWD2BBHKQ, seg_pos={0 0}, upload=517528]: Uploading piece... Dec 30 05:50:09.727263063 (590h21m9.598346108s) [upload=517528]: Next returned nil segment Dec 30 05:50:09.727283351 (590h21m9.598366386s) [upload=517528]: Waiting for error group managing segments... Dec 30 05:50:10.010280438 (590h21m9.881363483s) [noise=true , address=145.79.227.5:4223 , piece_id=X6CJNG227YK7H3FGKWKBOKSSPH44TZ5FEV2EHVHPALB7Y7CNP3AA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.015210825 (590h21m9.88629387s) [noise=true , address=145.79.227.48:4302 , piece_id=H6HID65U74FXC7VS5E52IPRC35DLKFCKZOUO3QGDU5CXHLMVVBPA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.018545656 (590h21m9.889628701s) [noise=true , address=145.79.227.9:4367 , piece_id=QHJ337HTW2NO3MXCITYPPE443FKHLPL5IG5N6BXIBPSJJWV6LTOQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.043972091 (590h21m9.915055137s) [noise=true , address=145.79.227.16:4910 , piece_id=A4I62LSDXLILZECT5CPFSDBZRZLXY4IHZKDJUETACHGXXJJNCUNA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.047623925 (590h21m9.91870697s) [noise=true , address=145.79.227.23:4545 , piece_id=MZXZHBGJEW6HQM7V2C55QGQIAEBI5ONN3MWWYMTHHPQBRJRQZNPQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.102492926 (590h21m9.973575971s) [noise=true , address=145.79.227.15:4875 , piece_id=PBOBRWD3KAFZJYTWVND3Y36RO5XY7RVIYDCR6RV55NWPLBGGZCBQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.214049859 (590h21m10.085132894s) [noise=true , address=77.90.31.16:4735 , piece_id=WK6DWLTVPXCGINS5QBSCG6CHYXUDMM36ICUPG7TMF2F7XXTX6CIA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.311122282 (590h21m10.182205327s) [noise=true , address=77.90.31.15:4700 , piece_id=OMFZWUMEDLVNDKATUIXFLZUKCDR3PQRI7AZ4JFK75QQ7N22SWEVQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.327549262 (590h21m10.198632297s) [noise=true , address=77.90.31.11:4555 , piece_id=U2GJ7KI4S6RHCDJWNOEJXAU4NV7V3ODOF3BGNICVIZLBDHF3H7JQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.356830358 (590h21m10.227913403s) [noise=true , address=45.91.116.12:4226 , piece_id=2VHV7LSCJQA3L4TCDEODEAQL3J2MFJ2U23D6UML7N7TQRUSYVB4Q, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.419134055 (590h21m10.290217101s) [noise=true , address=45.91.116.4:4940 , piece_id=SHHV2UXBGHGDSY5K46NIWCMN7FA3OW66C5BVKNTLJI2TGTUAVQCQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:10.481996056 (590h21m10.353079101s) [noise=true , address=45.91.116.11:4120 , piece_id=ZQXA4J2LZLFAJM2CPJNID267RGWBVN6D4OAZ44OXM43WKTBFV2UA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:11.021539821 (590h21m10.892622856s) [noise=true , address=77.90.31.13:4627 , piece_id=T3S7QE7W2SJV7IOLF3SH3PZA3ARIGT5FZPDYC7GN3MKM6XTB5GPA, seg_pos={0 0}, upload=517528]: Done uploading piece. err: <nil> Dec 30 05:50:11.021569967 (590h21m10.892653012s) [seg_pos={0 0}, upload=517528]: Segment reached optimal threshold of 13 pieces. Dec 30 05:50:11.021789777 (590h21m10.892872822s) [noise=true , address=45.91.116.7:4010 , piece_id=RMXZNAEVMZZ2KPBZKSYQLEZAKAFRKH4LJ55TQLT6XXXSUVRRQBIQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 30 05:50:11.021826537 (590h21m10.892909582s) [noise=true , address=45.91.116.7:4010 , piece_id=RMXZNAEVMZZ2KPBZKSYQLEZAKAFRKH4LJ55TQLT6XXXSUVRRQBIQ, seg_pos={0 0}, upload=517528]: Long tail cancellation due to optimal threshold Dec 30 05:50:11.021899866 (590h21m10.892982911s) [noise=true , address=45.91.116.8:4048 , piece_id=HPQ2ZE57MQ2ZRUQZ3B5SWUPENFTEAXTM3U3ECRWBGEQLWD2BBHKQ, seg_pos={0 0}, upload=517528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 30 05:50:11.021908542 (590h21m10.892991577s) [noise=true , address=45.91.116.8:4048 , piece_id=HPQ2ZE57MQ2ZRUQZ3B5SWUPENFTEAXTM3U3ECRWBGEQLWD2BBHKQ, seg_pos={0 0}, upload=517528]: Long tail cancellation due to optimal threshold Dec 30 05:50:11.022066914 (590h21m10.893149949s) [noise=true , address=45.91.116.13:4153 , piece_id=HXDPZTZ4SPH2MJBKC5PLGKZDOXKJOQZI3LGMU3BV6MGYIMKTEI7Q, seg_pos={0 0}, upload=517528]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 30 05:50:11.022074408 (590h21m10.893157443s) [noise=true , address=45.91.116.13:4153 , piece_id=HXDPZTZ4SPH2MJBKC5PLGKZDOXKJOQZI3LGMU3BV6MGYIMKTEI7Q, seg_pos={0 0}, upload=517528]: Long tail cancellation due to optimal threshold Dec 30 05:50:11.022094626 (590h21m10.893177671s) [seg_pos={0 0}, upload=517528]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:50:11.022160422 (590h21m10.893243457s) [upload=517528]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:50:11.022165092 (590h21m10.893248127s) [upload=517528]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:50:11.313202827 (590h21m11.184285872s) [upload=517528]: Done uploading segments... Dec 30 05:50:11.313228124 (590h21m11.18431117s) [upload=517528]: Upload finished. err: <nil> Duration: 30.003466953s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [X.X....X........X...........................................] duration: 3.554533358s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:50:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.555911873s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.XX.XXX...X.XXXXX..X..X...X....X.X.......X..X..X....X.XX.X..] duration: 6.058948307s trace: https://jaeger.dev.storj.io/trace/17c2f90cd6f7ddc5 FAIL (2025-12-30 05:49:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: PHWV3L3U4IFKAKGG5WTCDVI4Z7K2HSD3IVQDUD44HHJ2B3CX3VPQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: CGU6MR4W5BKLXSE4LOEKPFG2H2HZJBAPWR5POLHWZRA2J6DIT3JQ) context deadline exceeded ======LOGS====== ts... Dec 30 05:48:23.400444420 (590h19m23.271527465s) [upload=517501]: Got next segment. Inline: false Dec 30 05:48:23.400468146 (590h19m23.271551181s) [upload=517501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:48:23.400472213 (590h19m23.271555249s) [upload=517501]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:48:23.645573459 (590h19m23.516656514s) [seg_pos={0 0}, upload=517501]: Begin upload segment... Dec 30 05:48:23.645632932 (590h19m23.516715967s) [seg_pos={0 0}, upload=517501]: Done begin upload segment. Dec 30 05:48:23.645663440 (590h19m23.516746485s) [noise=true , address=145.79.227.40:4050 , piece_id=BVYF3CSRLHRMMBDKW7FMGG3VLOYQUMX5DH43ZOWCDWKY26FR6ZDQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.645685643 (590h19m23.516768678s) [noise=true , address=145.79.227.11:4731 , piece_id=LHZ4AJPSYZEPLMNBLVQBDMPXXQH3UGL2ZPJLCU2G3PHUGGECQ4PA, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.645729817 (590h19m23.516812862s) [noise=true , address=145.79.227.42:4122 , piece_id=2GQM47DP4IX3HJKB44TZARBFUJWEQJW4RGHJBVLAZZEH3KYEVGYQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.646333907 (590h19m23.517416972s) [noise=true , address=45.91.116.13:4153 , piece_id=BQZTNF4AW26DDSGHQN6BSB2GABSM34EHLHRMO7JYB4RWE34S5ZTA, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.646837697 (590h19m23.517920742s) [noise=true , address=145.79.227.9:4366 , piece_id=WXVVAEKL2MULYQKQWWOZSMEPKAUP5S3KOP6AMIZPAUH2OZXGHEGQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.647350894 (590h19m23.518433929s) [noise=true , address=45.91.116.9:4262 , piece_id=MML4YJZF34OSHKHSZWITZ6MNBLALZ5ZTE4UVSEUJYAWWMRDMZEIQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.647884790 (590h19m23.518967835s) [noise=true , address=145.79.227.38:4978 , piece_id=GFZUF24UMTSIOSZ5GENHUVUDAHLXPACBZVTEWKEXQLDORI634V5Q, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.648395473 (590h19m23.519478518s) [noise=true , address=145.79.227.37:4566 , piece_id=GR35CT3OJET5HK3FNE5KEFYWHNZ3ZYXOTK4QAGEKMZKKKF6GDRMA, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.648968935 (590h19m23.52005198s) [noise=true , address=77.90.31.16:4735 , piece_id=DG5ROQE4ZLDV6WWCZ5URJEHZPSINHUI6OLKY53EJGFWS353HEYDA, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.649748869 (590h19m23.520831914s) [noise=true , address=145.79.227.28:4725 , piece_id=5SYHRJXNOCM47KO4Y2DJACANFKTOMYUJAVVMRP6YVQ3ANXN5ON5A, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.651835563 (590h19m23.522918608s) [noise=true , address=45.91.116.8:4045 , piece_id=CGU6MR4W5BKLXSE4LOEKPFG2H2HZJBAPWR5POLHWZRA2J6DIT3JQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.652034151 (590h19m23.523117196s) [noise=true , address=45.91.116.4:4940 , piece_id=RHUCENOVB7OMXIPBWGFLPWUKQTNQ62IUT5BBZKK6S752C3AIB4EA, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.653951342 (590h19m23.525034377s) [noise=true , address=45.91.116.10:4083 , piece_id=C2AKCFJDJEM34WMC4NYEPUH5DTTTFGUMMAYP4E3DOLKCAMRDWTVQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.655946050 (590h19m23.527029095s) [noise=true , address=45.91.116.5:4298 , piece_id=OHNW75EMWYE6UBDG6ZSGQALWPCH5BYPMZG6GKXSSVAKLHZSHNN4A, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.657864404 (590h19m23.528947449s) [noise=true , address=77.90.31.10:4519 , piece_id=PHWV3L3U4IFKAKGG5WTCDVI4Z7K2HSD3IVQDUD44HHJ2B3CX3VPQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.660117183 (590h19m23.531200238s) [noise=true , address=145.79.227.52:4446 , piece_id=VU3PLQR4MHEQMSOJUKE2MNUZ6C2JGDUIYD2TS447YRXZBEZ7KTUQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:25.262924416 (590h19m25.134007461s) [upload=517501]: Next returned nil segment Dec 30 05:48:25.262942139 (590h19m25.134025184s) [upload=517501]: Waiting for error group managing segments... Dec 30 05:48:25.648085269 (590h19m25.519168304s) [noise=true , address=145.79.227.11:4731 , piece_id=LHZ4AJPSYZEPLMNBLVQBDMPXXQH3UGL2ZPJLCU2G3PHUGGECQ4PA, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.661369936 (590h19m25.532452971s) [noise=true , address=145.79.227.42:4122 , piece_id=2GQM47DP4IX3HJKB44TZARBFUJWEQJW4RGHJBVLAZZEH3KYEVGYQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.664894076 (590h19m25.535977111s) [noise=true , address=145.79.227.38:4978 , piece_id=GFZUF24UMTSIOSZ5GENHUVUDAHLXPACBZVTEWKEXQLDORI634V5Q, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.669483155 (590h19m25.54056619s) [noise=true , address=145.79.227.40:4050 , piece_id=BVYF3CSRLHRMMBDKW7FMGG3VLOYQUMX5DH43ZOWCDWKY26FR6ZDQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.672022339 (590h19m25.543105374s) [noise=true , address=145.79.227.28:4725 , piece_id=5SYHRJXNOCM47KO4Y2DJACANFKTOMYUJAVVMRP6YVQ3ANXN5ON5A, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.686349791 (590h19m25.557432826s) [noise=true , address=145.79.227.52:4446 , piece_id=VU3PLQR4MHEQMSOJUKE2MNUZ6C2JGDUIYD2TS447YRXZBEZ7KTUQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.731318389 (590h19m25.602401434s) [noise=true , address=145.79.227.9:4366 , piece_id=WXVVAEKL2MULYQKQWWOZSMEPKAUP5S3KOP6AMIZPAUH2OZXGHEGQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.745864798 (590h19m25.616947843s) [noise=true , address=145.79.227.37:4566 , piece_id=GR35CT3OJET5HK3FNE5KEFYWHNZ3ZYXOTK4QAGEKMZKKKF6GDRMA, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.909555964 (590h19m25.780639009s) [noise=true , address=77.90.31.10:4519 , piece_id=PHWV3L3U4IFKAKGG5WTCDVI4Z7K2HSD3IVQDUD44HHJ2B3CX3VPQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.980114600 (590h19m25.851197635s) [noise=true , address=45.91.116.4:4940 , piece_id=RHUCENOVB7OMXIPBWGFLPWUKQTNQ62IUT5BBZKK6S752C3AIB4EA, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.992152853 (590h19m25.863235888s) [noise=true , address=45.91.116.8:4045 , piece_id=CGU6MR4W5BKLXSE4LOEKPFG2H2HZJBAPWR5POLHWZRA2J6DIT3JQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:26.117194033 (590h19m25.988277068s) [noise=true , address=45.91.116.9:4262 , piece_id=MML4YJZF34OSHKHSZWITZ6MNBLALZ5ZTE4UVSEUJYAWWMRDMZEIQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:26.878543846 (590h19m26.749626881s) [noise=true , address=45.91.116.5:4298 , piece_id=OHNW75EMWYE6UBDG6ZSGQALWPCH5BYPMZG6GKXSSVAKLHZSHNN4A, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:26.878570396 (590h19m26.749653441s) [seg_pos={0 0}, upload=517501]: Segment reached optimal threshold of 13 pieces. Dec 30 05:48:26.878780306 (590h19m26.749863341s) [noise=true , address=45.91.116.13:4153 , piece_id=BQZTNF4AW26DDSGHQN6BSB2GABSM34EHLHRMO7JYB4RWE34S5ZTA, seg_pos={0 0}, upload=517501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:34476->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:34476->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34476->45.91.116.13:4153: use of closed network connection Dec 30 05:48:26.878790736 (590h19m26.749873781s) [noise=true , address=45.91.116.13:4153 , piece_id=BQZTNF4AW26DDSGHQN6BSB2GABSM34EHLHRMO7JYB4RWE34S5ZTA, seg_pos={0 0}, upload=517501]: Long tail cancellation due to optimal threshold Dec 30 05:48:26.878873994 (590h19m26.749957029s) [noise=true , address=45.91.116.10:4083 , piece_id=C2AKCFJDJEM34WMC4NYEPUH5DTTTFGUMMAYP4E3DOLKCAMRDWTVQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:42784->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:42784->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42784->45.91.116.10:4083: use of closed network connection Dec 30 05:48:26.878882190 (590h19m26.749965225s) [noise=true , address=45.91.116.10:4083 , piece_id=C2AKCFJDJEM34WMC4NYEPUH5DTTTFGUMMAYP4E3DOLKCAMRDWTVQ, seg_pos={0 0}, upload=517501]: Long tail cancellation due to optimal threshold Dec 30 05:48:26.879091569 (590h19m26.750174614s) [noise=true , address=77.90.31.16:4735 , piece_id=DG5ROQE4ZLDV6WWCZ5URJEHZPSINHUI6OLKY53EJGFWS353HEYDA, seg_pos={0 0}, upload=517501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 30 05:48:26.879098903 (590h19m26.750181948s) [noise=true , address=77.90.31.16:4735 , piece_id=DG5ROQE4ZLDV6WWCZ5URJEHZPSINHUI6OLKY53EJGFWS353HEYDA, seg_pos={0 0}, upload=517501]: Long tail cancellation due to optimal threshold Dec 30 05:48:26.879116305 (590h19m26.75019934s) [seg_pos={0 0}, upload=517501]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:48:26.879227087 (590h19m26.750310132s) [upload=517501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:48:26.879234661 (590h19m26.750317716s) [upload=517501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:48:27.166158644 (590h19m27.037241679s) [upload=517501]: Done uploading segments... Dec 30 05:48:27.166176999 (590h19m27.037260044s) [upload=517501]: Upload finished. err: <nil> Duration: 6.058974018s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.X.X....X........X..........................................] duration: 3.533406098s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:49:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.533421106s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..XX.XXX...X.XXXXX..X..X...X....X.X.......X..X..X....X.XX.X.] duration: 5.803523916s trace: https://jaeger.dev.storj.io/trace/17c2f90cd6f7ddc5 FAIL (2025-12-30 05:49:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: PHWV3L3U4IFKAKGG5WTCDVI4Z7K2HSD3IVQDUD44HHJ2B3CX3VPQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: CGU6MR4W5BKLXSE4LOEKPFG2H2HZJBAPWR5POLHWZRA2J6DIT3JQ) context deadline exceeded ======LOGS====== ts... Dec 30 05:48:23.400444420 (590h19m23.271527465s) [upload=517501]: Got next segment. Inline: false Dec 30 05:48:23.400468146 (590h19m23.271551181s) [upload=517501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:48:23.400472213 (590h19m23.271555249s) [upload=517501]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:48:23.645573459 (590h19m23.516656514s) [seg_pos={0 0}, upload=517501]: Begin upload segment... Dec 30 05:48:23.645632932 (590h19m23.516715967s) [seg_pos={0 0}, upload=517501]: Done begin upload segment. Dec 30 05:48:23.645663440 (590h19m23.516746485s) [noise=true , address=145.79.227.40:4050 , piece_id=BVYF3CSRLHRMMBDKW7FMGG3VLOYQUMX5DH43ZOWCDWKY26FR6ZDQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.645685643 (590h19m23.516768678s) [noise=true , address=145.79.227.11:4731 , piece_id=LHZ4AJPSYZEPLMNBLVQBDMPXXQH3UGL2ZPJLCU2G3PHUGGECQ4PA, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.645729817 (590h19m23.516812862s) [noise=true , address=145.79.227.42:4122 , piece_id=2GQM47DP4IX3HJKB44TZARBFUJWEQJW4RGHJBVLAZZEH3KYEVGYQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.646333907 (590h19m23.517416972s) [noise=true , address=45.91.116.13:4153 , piece_id=BQZTNF4AW26DDSGHQN6BSB2GABSM34EHLHRMO7JYB4RWE34S5ZTA, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.646837697 (590h19m23.517920742s) [noise=true , address=145.79.227.9:4366 , piece_id=WXVVAEKL2MULYQKQWWOZSMEPKAUP5S3KOP6AMIZPAUH2OZXGHEGQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.647350894 (590h19m23.518433929s) [noise=true , address=45.91.116.9:4262 , piece_id=MML4YJZF34OSHKHSZWITZ6MNBLALZ5ZTE4UVSEUJYAWWMRDMZEIQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.647884790 (590h19m23.518967835s) [noise=true , address=145.79.227.38:4978 , piece_id=GFZUF24UMTSIOSZ5GENHUVUDAHLXPACBZVTEWKEXQLDORI634V5Q, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.648395473 (590h19m23.519478518s) [noise=true , address=145.79.227.37:4566 , piece_id=GR35CT3OJET5HK3FNE5KEFYWHNZ3ZYXOTK4QAGEKMZKKKF6GDRMA, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.648968935 (590h19m23.52005198s) [noise=true , address=77.90.31.16:4735 , piece_id=DG5ROQE4ZLDV6WWCZ5URJEHZPSINHUI6OLKY53EJGFWS353HEYDA, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.649748869 (590h19m23.520831914s) [noise=true , address=145.79.227.28:4725 , piece_id=5SYHRJXNOCM47KO4Y2DJACANFKTOMYUJAVVMRP6YVQ3ANXN5ON5A, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.651835563 (590h19m23.522918608s) [noise=true , address=45.91.116.8:4045 , piece_id=CGU6MR4W5BKLXSE4LOEKPFG2H2HZJBAPWR5POLHWZRA2J6DIT3JQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.652034151 (590h19m23.523117196s) [noise=true , address=45.91.116.4:4940 , piece_id=RHUCENOVB7OMXIPBWGFLPWUKQTNQ62IUT5BBZKK6S752C3AIB4EA, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.653951342 (590h19m23.525034377s) [noise=true , address=45.91.116.10:4083 , piece_id=C2AKCFJDJEM34WMC4NYEPUH5DTTTFGUMMAYP4E3DOLKCAMRDWTVQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.655946050 (590h19m23.527029095s) [noise=true , address=45.91.116.5:4298 , piece_id=OHNW75EMWYE6UBDG6ZSGQALWPCH5BYPMZG6GKXSSVAKLHZSHNN4A, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.657864404 (590h19m23.528947449s) [noise=true , address=77.90.31.10:4519 , piece_id=PHWV3L3U4IFKAKGG5WTCDVI4Z7K2HSD3IVQDUD44HHJ2B3CX3VPQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.660117183 (590h19m23.531200238s) [noise=true , address=145.79.227.52:4446 , piece_id=VU3PLQR4MHEQMSOJUKE2MNUZ6C2JGDUIYD2TS447YRXZBEZ7KTUQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:25.262924416 (590h19m25.134007461s) [upload=517501]: Next returned nil segment Dec 30 05:48:25.262942139 (590h19m25.134025184s) [upload=517501]: Waiting for error group managing segments... Dec 30 05:48:25.648085269 (590h19m25.519168304s) [noise=true , address=145.79.227.11:4731 , piece_id=LHZ4AJPSYZEPLMNBLVQBDMPXXQH3UGL2ZPJLCU2G3PHUGGECQ4PA, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.661369936 (590h19m25.532452971s) [noise=true , address=145.79.227.42:4122 , piece_id=2GQM47DP4IX3HJKB44TZARBFUJWEQJW4RGHJBVLAZZEH3KYEVGYQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.664894076 (590h19m25.535977111s) [noise=true , address=145.79.227.38:4978 , piece_id=GFZUF24UMTSIOSZ5GENHUVUDAHLXPACBZVTEWKEXQLDORI634V5Q, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.669483155 (590h19m25.54056619s) [noise=true , address=145.79.227.40:4050 , piece_id=BVYF3CSRLHRMMBDKW7FMGG3VLOYQUMX5DH43ZOWCDWKY26FR6ZDQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.672022339 (590h19m25.543105374s) [noise=true , address=145.79.227.28:4725 , piece_id=5SYHRJXNOCM47KO4Y2DJACANFKTOMYUJAVVMRP6YVQ3ANXN5ON5A, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.686349791 (590h19m25.557432826s) [noise=true , address=145.79.227.52:4446 , piece_id=VU3PLQR4MHEQMSOJUKE2MNUZ6C2JGDUIYD2TS447YRXZBEZ7KTUQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.731318389 (590h19m25.602401434s) [noise=true , address=145.79.227.9:4366 , piece_id=WXVVAEKL2MULYQKQWWOZSMEPKAUP5S3KOP6AMIZPAUH2OZXGHEGQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.745864798 (590h19m25.616947843s) [noise=true , address=145.79.227.37:4566 , piece_id=GR35CT3OJET5HK3FNE5KEFYWHNZ3ZYXOTK4QAGEKMZKKKF6GDRMA, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.909555964 (590h19m25.780639009s) [noise=true , address=77.90.31.10:4519 , piece_id=PHWV3L3U4IFKAKGG5WTCDVI4Z7K2HSD3IVQDUD44HHJ2B3CX3VPQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.980114600 (590h19m25.851197635s) [noise=true , address=45.91.116.4:4940 , piece_id=RHUCENOVB7OMXIPBWGFLPWUKQTNQ62IUT5BBZKK6S752C3AIB4EA, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.992152853 (590h19m25.863235888s) [noise=true , address=45.91.116.8:4045 , piece_id=CGU6MR4W5BKLXSE4LOEKPFG2H2HZJBAPWR5POLHWZRA2J6DIT3JQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:26.117194033 (590h19m25.988277068s) [noise=true , address=45.91.116.9:4262 , piece_id=MML4YJZF34OSHKHSZWITZ6MNBLALZ5ZTE4UVSEUJYAWWMRDMZEIQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:26.878543846 (590h19m26.749626881s) [noise=true , address=45.91.116.5:4298 , piece_id=OHNW75EMWYE6UBDG6ZSGQALWPCH5BYPMZG6GKXSSVAKLHZSHNN4A, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:26.878570396 (590h19m26.749653441s) [seg_pos={0 0}, upload=517501]: Segment reached optimal threshold of 13 pieces. Dec 30 05:48:26.878780306 (590h19m26.749863341s) [noise=true , address=45.91.116.13:4153 , piece_id=BQZTNF4AW26DDSGHQN6BSB2GABSM34EHLHRMO7JYB4RWE34S5ZTA, seg_pos={0 0}, upload=517501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:34476->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:34476->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34476->45.91.116.13:4153: use of closed network connection Dec 30 05:48:26.878790736 (590h19m26.749873781s) [noise=true , address=45.91.116.13:4153 , piece_id=BQZTNF4AW26DDSGHQN6BSB2GABSM34EHLHRMO7JYB4RWE34S5ZTA, seg_pos={0 0}, upload=517501]: Long tail cancellation due to optimal threshold Dec 30 05:48:26.878873994 (590h19m26.749957029s) [noise=true , address=45.91.116.10:4083 , piece_id=C2AKCFJDJEM34WMC4NYEPUH5DTTTFGUMMAYP4E3DOLKCAMRDWTVQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:42784->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:42784->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42784->45.91.116.10:4083: use of closed network connection Dec 30 05:48:26.878882190 (590h19m26.749965225s) [noise=true , address=45.91.116.10:4083 , piece_id=C2AKCFJDJEM34WMC4NYEPUH5DTTTFGUMMAYP4E3DOLKCAMRDWTVQ, seg_pos={0 0}, upload=517501]: Long tail cancellation due to optimal threshold Dec 30 05:48:26.879091569 (590h19m26.750174614s) [noise=true , address=77.90.31.16:4735 , piece_id=DG5ROQE4ZLDV6WWCZ5URJEHZPSINHUI6OLKY53EJGFWS353HEYDA, seg_pos={0 0}, upload=517501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 30 05:48:26.879098903 (590h19m26.750181948s) [noise=true , address=77.90.31.16:4735 , piece_id=DG5ROQE4ZLDV6WWCZ5URJEHZPSINHUI6OLKY53EJGFWS353HEYDA, seg_pos={0 0}, upload=517501]: Long tail cancellation due to optimal threshold Dec 30 05:48:26.879116305 (590h19m26.75019934s) [seg_pos={0 0}, upload=517501]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:48:26.879227087 (590h19m26.750310132s) [upload=517501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:48:26.879234661 (590h19m26.750317716s) [upload=517501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:48:27.166158644 (590h19m27.037241679s) [upload=517501]: Done uploading segments... Dec 30 05:48:27.166176999 (590h19m27.037260044s) [upload=517501]: Upload finished. err: <nil> Duration: 5.803553152s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..X.X....X........X.........................................] duration: 4.337885736s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:48:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.337914942s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X..XX.XXX...X.XXXXX..X..X...X....X.X.......X..X..X....X.XX.X] duration: 30.00377436s trace: https://jaeger.dev.storj.io/trace/17c2f90cd6f7ddc5 FAIL (2025-12-30 05:48:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: PHWV3L3U4IFKAKGG5WTCDVI4Z7K2HSD3IVQDUD44HHJ2B3CX3VPQ) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: CGU6MR4W5BKLXSE4LOEKPFG2H2HZJBAPWR5POLHWZRA2J6DIT3JQ) context deadline exceeded ======LOGS====== ts... Dec 30 05:48:23.400444420 (590h19m23.271527465s) [upload=517501]: Got next segment. Inline: false Dec 30 05:48:23.400468146 (590h19m23.271551181s) [upload=517501]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:48:23.400472213 (590h19m23.271555249s) [upload=517501]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:48:23.645573459 (590h19m23.516656514s) [seg_pos={0 0}, upload=517501]: Begin upload segment... Dec 30 05:48:23.645632932 (590h19m23.516715967s) [seg_pos={0 0}, upload=517501]: Done begin upload segment. Dec 30 05:48:23.645663440 (590h19m23.516746485s) [noise=true , address=145.79.227.40:4050 , piece_id=BVYF3CSRLHRMMBDKW7FMGG3VLOYQUMX5DH43ZOWCDWKY26FR6ZDQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.645685643 (590h19m23.516768678s) [noise=true , address=145.79.227.11:4731 , piece_id=LHZ4AJPSYZEPLMNBLVQBDMPXXQH3UGL2ZPJLCU2G3PHUGGECQ4PA, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.645729817 (590h19m23.516812862s) [noise=true , address=145.79.227.42:4122 , piece_id=2GQM47DP4IX3HJKB44TZARBFUJWEQJW4RGHJBVLAZZEH3KYEVGYQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.646333907 (590h19m23.517416972s) [noise=true , address=45.91.116.13:4153 , piece_id=BQZTNF4AW26DDSGHQN6BSB2GABSM34EHLHRMO7JYB4RWE34S5ZTA, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.646837697 (590h19m23.517920742s) [noise=true , address=145.79.227.9:4366 , piece_id=WXVVAEKL2MULYQKQWWOZSMEPKAUP5S3KOP6AMIZPAUH2OZXGHEGQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.647350894 (590h19m23.518433929s) [noise=true , address=45.91.116.9:4262 , piece_id=MML4YJZF34OSHKHSZWITZ6MNBLALZ5ZTE4UVSEUJYAWWMRDMZEIQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.647884790 (590h19m23.518967835s) [noise=true , address=145.79.227.38:4978 , piece_id=GFZUF24UMTSIOSZ5GENHUVUDAHLXPACBZVTEWKEXQLDORI634V5Q, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.648395473 (590h19m23.519478518s) [noise=true , address=145.79.227.37:4566 , piece_id=GR35CT3OJET5HK3FNE5KEFYWHNZ3ZYXOTK4QAGEKMZKKKF6GDRMA, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.648968935 (590h19m23.52005198s) [noise=true , address=77.90.31.16:4735 , piece_id=DG5ROQE4ZLDV6WWCZ5URJEHZPSINHUI6OLKY53EJGFWS353HEYDA, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.649748869 (590h19m23.520831914s) [noise=true , address=145.79.227.28:4725 , piece_id=5SYHRJXNOCM47KO4Y2DJACANFKTOMYUJAVVMRP6YVQ3ANXN5ON5A, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.651835563 (590h19m23.522918608s) [noise=true , address=45.91.116.8:4045 , piece_id=CGU6MR4W5BKLXSE4LOEKPFG2H2HZJBAPWR5POLHWZRA2J6DIT3JQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.652034151 (590h19m23.523117196s) [noise=true , address=45.91.116.4:4940 , piece_id=RHUCENOVB7OMXIPBWGFLPWUKQTNQ62IUT5BBZKK6S752C3AIB4EA, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.653951342 (590h19m23.525034377s) [noise=true , address=45.91.116.10:4083 , piece_id=C2AKCFJDJEM34WMC4NYEPUH5DTTTFGUMMAYP4E3DOLKCAMRDWTVQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.655946050 (590h19m23.527029095s) [noise=true , address=45.91.116.5:4298 , piece_id=OHNW75EMWYE6UBDG6ZSGQALWPCH5BYPMZG6GKXSSVAKLHZSHNN4A, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.657864404 (590h19m23.528947449s) [noise=true , address=77.90.31.10:4519 , piece_id=PHWV3L3U4IFKAKGG5WTCDVI4Z7K2HSD3IVQDUD44HHJ2B3CX3VPQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:23.660117183 (590h19m23.531200238s) [noise=true , address=145.79.227.52:4446 , piece_id=VU3PLQR4MHEQMSOJUKE2MNUZ6C2JGDUIYD2TS447YRXZBEZ7KTUQ, seg_pos={0 0}, upload=517501]: Uploading piece... Dec 30 05:48:25.262924416 (590h19m25.134007461s) [upload=517501]: Next returned nil segment Dec 30 05:48:25.262942139 (590h19m25.134025184s) [upload=517501]: Waiting for error group managing segments... Dec 30 05:48:25.648085269 (590h19m25.519168304s) [noise=true , address=145.79.227.11:4731 , piece_id=LHZ4AJPSYZEPLMNBLVQBDMPXXQH3UGL2ZPJLCU2G3PHUGGECQ4PA, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.661369936 (590h19m25.532452971s) [noise=true , address=145.79.227.42:4122 , piece_id=2GQM47DP4IX3HJKB44TZARBFUJWEQJW4RGHJBVLAZZEH3KYEVGYQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.664894076 (590h19m25.535977111s) [noise=true , address=145.79.227.38:4978 , piece_id=GFZUF24UMTSIOSZ5GENHUVUDAHLXPACBZVTEWKEXQLDORI634V5Q, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.669483155 (590h19m25.54056619s) [noise=true , address=145.79.227.40:4050 , piece_id=BVYF3CSRLHRMMBDKW7FMGG3VLOYQUMX5DH43ZOWCDWKY26FR6ZDQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.672022339 (590h19m25.543105374s) [noise=true , address=145.79.227.28:4725 , piece_id=5SYHRJXNOCM47KO4Y2DJACANFKTOMYUJAVVMRP6YVQ3ANXN5ON5A, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.686349791 (590h19m25.557432826s) [noise=true , address=145.79.227.52:4446 , piece_id=VU3PLQR4MHEQMSOJUKE2MNUZ6C2JGDUIYD2TS447YRXZBEZ7KTUQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.731318389 (590h19m25.602401434s) [noise=true , address=145.79.227.9:4366 , piece_id=WXVVAEKL2MULYQKQWWOZSMEPKAUP5S3KOP6AMIZPAUH2OZXGHEGQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.745864798 (590h19m25.616947843s) [noise=true , address=145.79.227.37:4566 , piece_id=GR35CT3OJET5HK3FNE5KEFYWHNZ3ZYXOTK4QAGEKMZKKKF6GDRMA, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.909555964 (590h19m25.780639009s) [noise=true , address=77.90.31.10:4519 , piece_id=PHWV3L3U4IFKAKGG5WTCDVI4Z7K2HSD3IVQDUD44HHJ2B3CX3VPQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.980114600 (590h19m25.851197635s) [noise=true , address=45.91.116.4:4940 , piece_id=RHUCENOVB7OMXIPBWGFLPWUKQTNQ62IUT5BBZKK6S752C3AIB4EA, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:25.992152853 (590h19m25.863235888s) [noise=true , address=45.91.116.8:4045 , piece_id=CGU6MR4W5BKLXSE4LOEKPFG2H2HZJBAPWR5POLHWZRA2J6DIT3JQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:26.117194033 (590h19m25.988277068s) [noise=true , address=45.91.116.9:4262 , piece_id=MML4YJZF34OSHKHSZWITZ6MNBLALZ5ZTE4UVSEUJYAWWMRDMZEIQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:26.878543846 (590h19m26.749626881s) [noise=true , address=45.91.116.5:4298 , piece_id=OHNW75EMWYE6UBDG6ZSGQALWPCH5BYPMZG6GKXSSVAKLHZSHNN4A, seg_pos={0 0}, upload=517501]: Done uploading piece. err: <nil> Dec 30 05:48:26.878570396 (590h19m26.749653441s) [seg_pos={0 0}, upload=517501]: Segment reached optimal threshold of 13 pieces. Dec 30 05:48:26.878780306 (590h19m26.749863341s) [noise=true , address=45.91.116.13:4153 , piece_id=BQZTNF4AW26DDSGHQN6BSB2GABSM34EHLHRMO7JYB4RWE34S5ZTA, seg_pos={0 0}, upload=517501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: write tcp 188.166.225.170:34476->45.91.116.13:4153: use of closed network connection; write tcp 188.166.225.170:34476->45.91.116.13:4153: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34476->45.91.116.13:4153: use of closed network connection Dec 30 05:48:26.878790736 (590h19m26.749873781s) [noise=true , address=45.91.116.13:4153 , piece_id=BQZTNF4AW26DDSGHQN6BSB2GABSM34EHLHRMO7JYB4RWE34S5ZTA, seg_pos={0 0}, upload=517501]: Long tail cancellation due to optimal threshold Dec 30 05:48:26.878873994 (590h19m26.749957029s) [noise=true , address=45.91.116.10:4083 , piece_id=C2AKCFJDJEM34WMC4NYEPUH5DTTTFGUMMAYP4E3DOLKCAMRDWTVQ, seg_pos={0 0}, upload=517501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:42784->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:42784->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42784->45.91.116.10:4083: use of closed network connection Dec 30 05:48:26.878882190 (590h19m26.749965225s) [noise=true , address=45.91.116.10:4083 , piece_id=C2AKCFJDJEM34WMC4NYEPUH5DTTTFGUMMAYP4E3DOLKCAMRDWTVQ, seg_pos={0 0}, upload=517501]: Long tail cancellation due to optimal threshold Dec 30 05:48:26.879091569 (590h19m26.750174614s) [noise=true , address=77.90.31.16:4735 , piece_id=DG5ROQE4ZLDV6WWCZ5URJEHZPSINHUI6OLKY53EJGFWS353HEYDA, seg_pos={0 0}, upload=517501]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 30 05:48:26.879098903 (590h19m26.750181948s) [noise=true , address=77.90.31.16:4735 , piece_id=DG5ROQE4ZLDV6WWCZ5URJEHZPSINHUI6OLKY53EJGFWS353HEYDA, seg_pos={0 0}, upload=517501]: Long tail cancellation due to optimal threshold Dec 30 05:48:26.879116305 (590h19m26.75019934s) [seg_pos={0 0}, upload=517501]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:48:26.879227087 (590h19m26.750310132s) [upload=517501]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:48:26.879234661 (590h19m26.750317716s) [upload=517501]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:48:27.166158644 (590h19m27.037241679s) [upload=517501]: Done uploading segments... Dec 30 05:48:27.166176999 (590h19m27.037260044s) [upload=517501]: Upload finished. err: <nil> Duration: 30.003799146s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...X.X....X........X........................................] duration: 3.426454372s trace: https://jaeger.dev.storj.io/trace/3457119f684cb9d FAIL (2025-12-30 05:48:11 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.426483058s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X..XX.XXX...X.XXXXX..X..X...X....X.X.......X..X..X....X.XX.] duration: 6.031374825s trace: https://jaeger.dev.storj.io/trace/50de85881f962460 FAIL (2025-12-30 05:47:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: JRNWPCP6CNASTPD5YKDIAZHR2HIDOGEVZ6NMX47GX3AP46JNYHHQ) context deadline exceeded; piecestore: (Node ID: 1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, Piece ID: UHBQJES6EU7QEKTVC4SKCX6BAFDVYDAEX76SY4Q5PBOPYOZ25THA) context deadline exceeded ======LOGS====== 0 05:46:53.012795453 (590h17m52.883878498s) [upload=517478]: Starting upload Dec 30 05:46:53.012882198 (590h17m52.883965243s) [upload=517478]: Done starting upload Dec 30 05:46:53.012986818 (590h17m52.884069863s) [upload=517478]: Uploading segments... Dec 30 05:46:53.013325974 (590h17m52.884409019s) [upload=517478]: Got next segment. Inline: false Dec 30 05:46:53.013424400 (590h17m52.884507445s) [upload=517478]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 05:46:53.013432156 (590h17m52.884515201s) [upload=517478]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 05:46:53.257491908 (590h17m53.128574943s) [seg_pos={0 0}, upload=517478]: Begin upload segment... Dec 30 05:46:53.257600905 (590h17m53.12868395s) [seg_pos={0 0}, upload=517478]: Done begin upload segment. Dec 30 05:46:53.257671419 (590h17m53.128754464s) [noise=true , address=145.79.227.25:4619 , piece_id=F3A7BNK5XWZALVBV6TUYG6XOYKBUXUHTUOTCMVIXOPX3VQ4774WQ, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.258797032 (590h17m53.129880077s) [noise=true , address=145.79.227.16:4911 , piece_id=FTR5ZERGFVC2JH7E63MCWIG22JH362HITH4XOJQA6OCXQGQXMAFA, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.258854192 (590h17m53.129937237s) [noise=true , address=145.79.227.46:4601 , piece_id=MHM5YPK32IBA67T74UCX555GFOSP53ALIZCMQZKJFYCISKKNDFGA, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.259449876 (590h17m53.130532921s) [noise=true , address=145.79.227.22:4510 , piece_id=VFHUJLFIGDQEZ6HQRQPVGJ35XRVSLCP3ENBQAONLNP6RP7GE66YQ, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.259491024 (590h17m53.130574069s) [noise=true , address=145.79.227.19:4079 , piece_id=NN474EKMGVWBWJRWIVQDLTLFZSV7MUJ4SJOHMRULEJSGRSAV23JA, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.259994784 (590h17m53.131077829s) [noise=true , address=145.79.227.38:4977 , piece_id=BIVMU7PGY5PJPOHNSRENK5B7QBB7ENYELP4RGZRX5COAI3CPYXDQ, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.260292610 (590h17m53.131375655s) [noise=true , address=145.79.227.2:4113 , piece_id=SQZ7C76CI7MLXIPC5S2UCFTQ6AENUYQ2ZWTPOS3OUJVLLVEBPUXQ, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.260621357 (590h17m53.131704402s) [noise=true , address=77.90.31.19:4841 , piece_id=JRNWPCP6CNASTPD5YKDIAZHR2HIDOGEVZ6NMX47GX3AP46JNYHHQ, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.260794997 (590h17m53.131878042s) [noise=true , address=145.79.227.37:4566 , piece_id=TIGQWMEIBJFTHS2YGWU3DDF4YZCJXSI3WTXRX7RE3C7IUBN3GIPA, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.261162797 (590h17m53.132245832s) [noise=true , address=77.90.31.11:4554 , piece_id=UHBQJES6EU7QEKTVC4SKCX6BAFDVYDAEX76SY4Q5PBOPYOZ25THA, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.261378388 (590h17m53.132461433s) [noise=true , address=145.79.227.32:4836 , piece_id=OVJSBZQUKTLU2YE4WFAQ5JQ6YG44QCZJEYG5OR7WE6MAAYRBER7A, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.261697545 (590h17m53.132780611s) [noise=true , address=45.91.116.12:4226 , piece_id=LU357TDY2ZUQ7TLSVWM62E6QEIUOOBHJUUC3KEL252763B4YFL6Q, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.262190164 (590h17m53.133273209s) [noise=true , address=145.79.227.45:4229 , piece_id=5ENZQXZHCNWASWMO775MJFTXF6TOOA6I7VD4MGAXMLFUOF6RH2RA, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.262402448 (590h17m53.133485483s) [noise=true , address=145.79.227.52:4446 , piece_id=5ISNGCHVSSME6EM7E3Q2SDTVEO4WF7VGHXMPLDA6MNHUAIFSGSFA, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.263016006 (590h17m53.134099051s) [noise=true , address=45.91.116.4:4939 , piece_id=GAANJKTDQOUC2CTTH2CMY4F3C2RH5DTZ5LCVWSRUFFKAJILWIHVQ, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:53.263558349 (590h17m53.134641384s) [noise=true , address=45.91.116.5:4298 , piece_id=WB4POGJM57MQF2DKG7BJ424LPHWUASJJC2OVR4A6UJ2S36EHV3VA, seg_pos={0 0}, upload=517478]: Uploading piece... Dec 30 05:46:54.965961831 (590h17m54.837044866s) [upload=517478]: Next returned nil segment Dec 30 05:46:54.965975406 (590h17m54.837058441s) [upload=517478]: Waiting for error group managing segments... Dec 30 05:46:55.245851740 (590h17m55.116934775s) [noise=true , address=145.79.227.25:4619 , piece_id=F3A7BNK5XWZALVBV6TUYG6XOYKBUXUHTUOTCMVIXOPX3VQ4774WQ, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.246456000 (590h17m55.117539045s) [noise=true , address=145.79.227.45:4229 , piece_id=5ENZQXZHCNWASWMO775MJFTXF6TOOA6I7VD4MGAXMLFUOF6RH2RA, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.254156313 (590h17m55.125239348s) [noise=true , address=145.79.227.52:4446 , piece_id=5ISNGCHVSSME6EM7E3Q2SDTVEO4WF7VGHXMPLDA6MNHUAIFSGSFA, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.269694441 (590h17m55.140777476s) [noise=true , address=145.79.227.37:4566 , piece_id=TIGQWMEIBJFTHS2YGWU3DDF4YZCJXSI3WTXRX7RE3C7IUBN3GIPA, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.270822809 (590h17m55.141905844s) [noise=true , address=145.79.227.22:4510 , piece_id=VFHUJLFIGDQEZ6HQRQPVGJ35XRVSLCP3ENBQAONLNP6RP7GE66YQ, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.273548709 (590h17m55.144631744s) [noise=true , address=145.79.227.46:4601 , piece_id=MHM5YPK32IBA67T74UCX555GFOSP53ALIZCMQZKJFYCISKKNDFGA, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.274473851 (590h17m55.145556896s) [noise=true , address=145.79.227.32:4836 , piece_id=OVJSBZQUKTLU2YE4WFAQ5JQ6YG44QCZJEYG5OR7WE6MAAYRBER7A, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.329920050 (590h17m55.201003086s) [noise=true , address=145.79.227.38:4977 , piece_id=BIVMU7PGY5PJPOHNSRENK5B7QBB7ENYELP4RGZRX5COAI3CPYXDQ, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.330106185 (590h17m55.20118924s) [noise=true , address=145.79.227.19:4079 , piece_id=NN474EKMGVWBWJRWIVQDLTLFZSV7MUJ4SJOHMRULEJSGRSAV23JA, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.331420558 (590h17m55.202503593s) [noise=true , address=145.79.227.2:4113 , piece_id=SQZ7C76CI7MLXIPC5S2UCFTQ6AENUYQ2ZWTPOS3OUJVLLVEBPUXQ, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.343200498 (590h17m55.214283543s) [noise=true , address=145.79.227.16:4911 , piece_id=FTR5ZERGFVC2JH7E63MCWIG22JH362HITH4XOJQA6OCXQGQXMAFA, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.431556082 (590h17m55.302639127s) [noise=true , address=77.90.31.11:4554 , piece_id=UHBQJES6EU7QEKTVC4SKCX6BAFDVYDAEX76SY4Q5PBOPYOZ25THA, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.562019638 (590h17m55.433102683s) [noise=true , address=77.90.31.19:4841 , piece_id=JRNWPCP6CNASTPD5YKDIAZHR2HIDOGEVZ6NMX47GX3AP46JNYHHQ, seg_pos={0 0}, upload=517478]: Done uploading piece. err: <nil> Dec 30 05:46:55.562039837 (590h17m55.433122872s) [seg_pos={0 0}, upload=517478]: Segment reached optimal threshold of 13 pieces. Dec 30 05:46:55.562216062 (590h17m55.433299097s) [noise=true , address=45.91.116.12:4226 , piece_id=LU357TDY2ZUQ7TLSVWM62E6QEIUOOBHJUUC3KEL252763B4YFL6Q, seg_pos={0 0}, upload=517478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Dec 30 05:46:55.562222053 (590h17m55.433305098s) [noise=true , address=45.91.116.12:4226 , piece_id=LU357TDY2ZUQ7TLSVWM62E6QEIUOOBHJUUC3KEL252763B4YFL6Q, seg_pos={0 0}, upload=517478]: Long tail cancellation due to optimal threshold Dec 30 05:46:55.562459466 (590h17m55.433542511s) [noise=true , address=45.91.116.5:4298 , piece_id=WB4POGJM57MQF2DKG7BJ424LPHWUASJJC2OVR4A6UJ2S36EHV3VA, seg_pos={0 0}, upload=517478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:57448->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:57448->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57448->45.91.116.5:4298: use of closed network connection Dec 30 05:46:55.562467961 (590h17m55.433551007s) [noise=true , address=45.91.116.5:4298 , piece_id=WB4POGJM57MQF2DKG7BJ424LPHWUASJJC2OVR4A6UJ2S36EHV3VA, seg_pos={0 0}, upload=517478]: Long tail cancellation due to optimal threshold Dec 30 05:46:55.562478351 (590h17m55.433561396s) [noise=true , address=45.91.116.4:4939 , piece_id=GAANJKTDQOUC2CTTH2CMY4F3C2RH5DTZ5LCVWSRUFFKAJILWIHVQ, seg_pos={0 0}, upload=517478]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 30 05:46:55.562485125 (590h17m55.43356816s) [noise=true , address=45.91.116.4:4939 , piece_id=GAANJKTDQOUC2CTTH2CMY4F3C2RH5DTZ5LCVWSRUFFKAJILWIHVQ, seg_pos={0 0}, upload=517478]: Long tail cancellation due to optimal threshold Dec 30 05:46:55.562502237 (590h17m55.433585282s) [seg_pos={0 0}, upload=517478]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 05:46:55.562568784 (590h17m55.433651819s) [upload=517478]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 05:46:55.562573392 (590h17m55.433656427s) [upload=517478]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 05:46:55.846330195 (590h17m55.71741323s) [upload=517478]: Done uploading segments... Dec 30 05:46:55.846408714 (590h17m55.717491749s) [upload=517478]: Upload finished. err: <nil> Duration: 6.031494963s core uplink us-select-1

1
...
11263
11264
11265
11266
11267
...
18121