Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 434723 reports (Page 8407 of 8695)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.XXX...X.........X.XXX......X...............................] duration: 5.494419659s FAIL (2025-07-09 03:22:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.494433244s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.765541446s FAIL (2025-07-09 03:22:08 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== expected piece hash; context canceled; EOF Jul 9 03:00:26.821547853 (377h50m1.592189189s) [noise=true , address=45.91.116.7:4010 , piece_id=OR5D2DCY5B4IKLDVVZT73DXDUTABGQ4K4VDJEL5HBSV3SQSS4GCQ, seg_pos={0 0}, upload=302277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Jul 9 03:00:26.821590575 (377h50m1.59223191s) [seg_pos={0 0}, upload=302277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 03:00:26.821670989 (377h50m1.592312314s) [upload=302277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 03:00:26.821675889 (377h50m1.592317224s) [upload=302277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 03:00:27.129149168 (377h50m1.899790543s) [upload=302277]: Done uploading segments... Jul 9 03:00:27.129166300 (377h50m1.899807675s) [upload=302277]: Upload finished. err: <nil> Jul 9 03:01:31.849644782 (377h51m6.620286117s) [upload=302293]: Starting upload Jul 9 03:01:31.849706480 (377h51m6.620347815s) [upload=302293]: Done starting upload Jul 9 03:01:31.849908567 (377h51m6.620549902s) [upload=302293]: Uploading segments... Jul 9 03:01:31.849940148 (377h51m6.620581493s) [upload=302293]: Got next segment. Inline: false Jul 9 03:01:31.849961679 (377h51m6.620603014s) [upload=302293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 03:01:31.849966318 (377h51m6.620607663s) [upload=302293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 03:01:32.082815894 (377h51m6.853457229s) [seg_pos={0 0}, upload=302293]: Begin upload segment... Jul 9 03:01:32.082857022 (377h51m6.853498357s) [seg_pos={0 0}, upload=302293]: Done begin upload segment. Jul 9 03:01:32.082877130 (377h51m6.853518476s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.082910053 (377h51m6.853551378s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083224024 (377h51m6.853865359s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083306551 (377h51m6.853947887s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083473250 (377h51m6.854114585s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083576027 (377h51m6.854217352s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083720604 (377h51m6.854361929s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083881712 (377h51m6.854523047s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084015208 (377h51m6.854656543s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084188880 (377h51m6.854830215s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084277228 (377h51m6.854918563s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084445321 (377h51m6.855086656s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084507439 (377h51m6.855148775s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084669670 (377h51m6.855311005s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084714255 (377h51m6.85535559s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084930158 (377h51m6.855571493s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:33.718884639 (377h51m8.489525974s) [upload=302293]: Next returned nil segment Jul 9 03:01:33.718901952 (377h51m8.489543287s) [upload=302293]: Waiting for error group managing segments... Jul 9 03:01:34.050524955 (377h51m8.82116629s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.058353302 (377h51m8.828994637s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.098151442 (377h51m8.868792777s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.110943775 (377h51m8.88158511s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.117461985 (377h51m8.88810332s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.161579608 (377h51m8.932220933s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.170088417 (377h51m8.940729752s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.170743521 (377h51m8.941384846s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.176248322 (377h51m8.946889647s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.183619354 (377h51m8.954260679s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.236599983 (377h51m9.007241318s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.242137387 (377h51m9.012778712s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.242995759 (377h51m9.013637094s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.243008334 (377h51m9.013649669s) [seg_pos={0 0}, upload=302293]: Segment reached optimal threshold of 13 pieces. Jul 9 03:01:34.243124386 (377h51m9.013765721s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243179531 (377h51m9.013820866s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243274663 (377h51m9.013915998s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243323757 (377h51m9.013965092s) [seg_pos={0 0}, upload=302293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 03:01:34.243412717 (377h51m9.014054052s) [upload=302293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 03:01:34.243417376 (377h51m9.014058711s) [upload=302293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 03:01:34.538397355 (377h51m9.30903869s) [upload=302293]: Done uploading segments... Jul 9 03:01:34.538417173 (377h51m9.309058508s) [upload=302293]: Upload finished. err: <nil> Duration: 5.765567085s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [..XXX...X.........X.XXX......X..............................] duration: 36.609830002s FAIL (2025-07-09 03:22:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 36.609853848s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X.....XXX.........X.X.....................................] duration: 6.101646203s FAIL (2025-07-09 03:21:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.101685028s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.73315563s FAIL (2025-07-09 03:21:11 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== expected piece hash; context canceled; EOF Jul 9 03:00:26.821547853 (377h50m1.592189189s) [noise=true , address=45.91.116.7:4010 , piece_id=OR5D2DCY5B4IKLDVVZT73DXDUTABGQ4K4VDJEL5HBSV3SQSS4GCQ, seg_pos={0 0}, upload=302277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Jul 9 03:00:26.821590575 (377h50m1.59223191s) [seg_pos={0 0}, upload=302277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 03:00:26.821670989 (377h50m1.592312314s) [upload=302277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 03:00:26.821675889 (377h50m1.592317224s) [upload=302277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 03:00:27.129149168 (377h50m1.899790543s) [upload=302277]: Done uploading segments... Jul 9 03:00:27.129166300 (377h50m1.899807675s) [upload=302277]: Upload finished. err: <nil> Jul 9 03:01:31.849644782 (377h51m6.620286117s) [upload=302293]: Starting upload Jul 9 03:01:31.849706480 (377h51m6.620347815s) [upload=302293]: Done starting upload Jul 9 03:01:31.849908567 (377h51m6.620549902s) [upload=302293]: Uploading segments... Jul 9 03:01:31.849940148 (377h51m6.620581493s) [upload=302293]: Got next segment. Inline: false Jul 9 03:01:31.849961679 (377h51m6.620603014s) [upload=302293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 03:01:31.849966318 (377h51m6.620607663s) [upload=302293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 03:01:32.082815894 (377h51m6.853457229s) [seg_pos={0 0}, upload=302293]: Begin upload segment... Jul 9 03:01:32.082857022 (377h51m6.853498357s) [seg_pos={0 0}, upload=302293]: Done begin upload segment. Jul 9 03:01:32.082877130 (377h51m6.853518476s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.082910053 (377h51m6.853551378s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083224024 (377h51m6.853865359s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083306551 (377h51m6.853947887s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083473250 (377h51m6.854114585s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083576027 (377h51m6.854217352s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083720604 (377h51m6.854361929s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083881712 (377h51m6.854523047s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084015208 (377h51m6.854656543s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084188880 (377h51m6.854830215s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084277228 (377h51m6.854918563s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084445321 (377h51m6.855086656s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084507439 (377h51m6.855148775s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084669670 (377h51m6.855311005s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084714255 (377h51m6.85535559s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084930158 (377h51m6.855571493s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:33.718884639 (377h51m8.489525974s) [upload=302293]: Next returned nil segment Jul 9 03:01:33.718901952 (377h51m8.489543287s) [upload=302293]: Waiting for error group managing segments... Jul 9 03:01:34.050524955 (377h51m8.82116629s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.058353302 (377h51m8.828994637s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.098151442 (377h51m8.868792777s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.110943775 (377h51m8.88158511s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.117461985 (377h51m8.88810332s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.161579608 (377h51m8.932220933s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.170088417 (377h51m8.940729752s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.170743521 (377h51m8.941384846s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.176248322 (377h51m8.946889647s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.183619354 (377h51m8.954260679s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.236599983 (377h51m9.007241318s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.242137387 (377h51m9.012778712s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.242995759 (377h51m9.013637094s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.243008334 (377h51m9.013649669s) [seg_pos={0 0}, upload=302293]: Segment reached optimal threshold of 13 pieces. Jul 9 03:01:34.243124386 (377h51m9.013765721s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243179531 (377h51m9.013820866s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243274663 (377h51m9.013915998s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243323757 (377h51m9.013965092s) [seg_pos={0 0}, upload=302293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 03:01:34.243412717 (377h51m9.014054052s) [upload=302293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 03:01:34.243417376 (377h51m9.014058711s) [upload=302293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 03:01:34.538397355 (377h51m9.30903869s) [upload=302293]: Done uploading segments... Jul 9 03:01:34.538417173 (377h51m9.309058508s) [upload=302293]: Upload finished. err: <nil> Duration: 5.733178724s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.969280326s FAIL (2025-07-09 03:20:50 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== expected piece hash; context canceled; EOF Jul 9 03:00:26.821547853 (377h50m1.592189189s) [noise=true , address=45.91.116.7:4010 , piece_id=OR5D2DCY5B4IKLDVVZT73DXDUTABGQ4K4VDJEL5HBSV3SQSS4GCQ, seg_pos={0 0}, upload=302277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Jul 9 03:00:26.821590575 (377h50m1.59223191s) [seg_pos={0 0}, upload=302277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 03:00:26.821670989 (377h50m1.592312314s) [upload=302277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 03:00:26.821675889 (377h50m1.592317224s) [upload=302277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 03:00:27.129149168 (377h50m1.899790543s) [upload=302277]: Done uploading segments... Jul 9 03:00:27.129166300 (377h50m1.899807675s) [upload=302277]: Upload finished. err: <nil> Jul 9 03:01:31.849644782 (377h51m6.620286117s) [upload=302293]: Starting upload Jul 9 03:01:31.849706480 (377h51m6.620347815s) [upload=302293]: Done starting upload Jul 9 03:01:31.849908567 (377h51m6.620549902s) [upload=302293]: Uploading segments... Jul 9 03:01:31.849940148 (377h51m6.620581493s) [upload=302293]: Got next segment. Inline: false Jul 9 03:01:31.849961679 (377h51m6.620603014s) [upload=302293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 03:01:31.849966318 (377h51m6.620607663s) [upload=302293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 03:01:32.082815894 (377h51m6.853457229s) [seg_pos={0 0}, upload=302293]: Begin upload segment... Jul 9 03:01:32.082857022 (377h51m6.853498357s) [seg_pos={0 0}, upload=302293]: Done begin upload segment. Jul 9 03:01:32.082877130 (377h51m6.853518476s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.082910053 (377h51m6.853551378s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083224024 (377h51m6.853865359s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083306551 (377h51m6.853947887s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083473250 (377h51m6.854114585s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083576027 (377h51m6.854217352s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083720604 (377h51m6.854361929s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083881712 (377h51m6.854523047s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084015208 (377h51m6.854656543s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084188880 (377h51m6.854830215s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084277228 (377h51m6.854918563s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084445321 (377h51m6.855086656s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084507439 (377h51m6.855148775s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084669670 (377h51m6.855311005s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084714255 (377h51m6.85535559s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084930158 (377h51m6.855571493s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:33.718884639 (377h51m8.489525974s) [upload=302293]: Next returned nil segment Jul 9 03:01:33.718901952 (377h51m8.489543287s) [upload=302293]: Waiting for error group managing segments... Jul 9 03:01:34.050524955 (377h51m8.82116629s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.058353302 (377h51m8.828994637s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.098151442 (377h51m8.868792777s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.110943775 (377h51m8.88158511s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.117461985 (377h51m8.88810332s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.161579608 (377h51m8.932220933s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.170088417 (377h51m8.940729752s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.170743521 (377h51m8.941384846s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.176248322 (377h51m8.946889647s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.183619354 (377h51m8.954260679s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.236599983 (377h51m9.007241318s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.242137387 (377h51m9.012778712s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.242995759 (377h51m9.013637094s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.243008334 (377h51m9.013649669s) [seg_pos={0 0}, upload=302293]: Segment reached optimal threshold of 13 pieces. Jul 9 03:01:34.243124386 (377h51m9.013765721s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243179531 (377h51m9.013820866s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243274663 (377h51m9.013915998s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243323757 (377h51m9.013965092s) [seg_pos={0 0}, upload=302293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 03:01:34.243412717 (377h51m9.014054052s) [upload=302293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 03:01:34.243417376 (377h51m9.014058711s) [upload=302293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 03:01:34.538397355 (377h51m9.30903869s) [upload=302293]: Done uploading segments... Jul 9 03:01:34.538417173 (377h51m9.309058508s) [upload=302293]: Upload finished. err: <nil> Duration: 5.969308029s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X.....XXX.........X.X....................................] duration: 7.451677521s FAIL (2025-07-09 03:20:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.451703641s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [X..XXX...X.........X.XXX......X.............................] duration: 9.436210636s FAIL (2025-07-09 03:20:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.436232798s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.755425417s FAIL (2025-07-09 03:20:34 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== expected piece hash; context canceled; EOF Jul 9 03:00:26.821547853 (377h50m1.592189189s) [noise=true , address=45.91.116.7:4010 , piece_id=OR5D2DCY5B4IKLDVVZT73DXDUTABGQ4K4VDJEL5HBSV3SQSS4GCQ, seg_pos={0 0}, upload=302277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Jul 9 03:00:26.821590575 (377h50m1.59223191s) [seg_pos={0 0}, upload=302277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 03:00:26.821670989 (377h50m1.592312314s) [upload=302277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 03:00:26.821675889 (377h50m1.592317224s) [upload=302277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 03:00:27.129149168 (377h50m1.899790543s) [upload=302277]: Done uploading segments... Jul 9 03:00:27.129166300 (377h50m1.899807675s) [upload=302277]: Upload finished. err: <nil> Jul 9 03:01:31.849644782 (377h51m6.620286117s) [upload=302293]: Starting upload Jul 9 03:01:31.849706480 (377h51m6.620347815s) [upload=302293]: Done starting upload Jul 9 03:01:31.849908567 (377h51m6.620549902s) [upload=302293]: Uploading segments... Jul 9 03:01:31.849940148 (377h51m6.620581493s) [upload=302293]: Got next segment. Inline: false Jul 9 03:01:31.849961679 (377h51m6.620603014s) [upload=302293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 03:01:31.849966318 (377h51m6.620607663s) [upload=302293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 03:01:32.082815894 (377h51m6.853457229s) [seg_pos={0 0}, upload=302293]: Begin upload segment... Jul 9 03:01:32.082857022 (377h51m6.853498357s) [seg_pos={0 0}, upload=302293]: Done begin upload segment. Jul 9 03:01:32.082877130 (377h51m6.853518476s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.082910053 (377h51m6.853551378s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083224024 (377h51m6.853865359s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083306551 (377h51m6.853947887s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083473250 (377h51m6.854114585s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083576027 (377h51m6.854217352s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083720604 (377h51m6.854361929s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083881712 (377h51m6.854523047s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084015208 (377h51m6.854656543s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084188880 (377h51m6.854830215s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084277228 (377h51m6.854918563s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084445321 (377h51m6.855086656s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084507439 (377h51m6.855148775s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084669670 (377h51m6.855311005s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084714255 (377h51m6.85535559s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084930158 (377h51m6.855571493s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:33.718884639 (377h51m8.489525974s) [upload=302293]: Next returned nil segment Jul 9 03:01:33.718901952 (377h51m8.489543287s) [upload=302293]: Waiting for error group managing segments... Jul 9 03:01:34.050524955 (377h51m8.82116629s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.058353302 (377h51m8.828994637s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.098151442 (377h51m8.868792777s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.110943775 (377h51m8.88158511s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.117461985 (377h51m8.88810332s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.161579608 (377h51m8.932220933s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.170088417 (377h51m8.940729752s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.170743521 (377h51m8.941384846s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.176248322 (377h51m8.946889647s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.183619354 (377h51m8.954260679s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.236599983 (377h51m9.007241318s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.242137387 (377h51m9.012778712s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.242995759 (377h51m9.013637094s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.243008334 (377h51m9.013649669s) [seg_pos={0 0}, upload=302293]: Segment reached optimal threshold of 13 pieces. Jul 9 03:01:34.243124386 (377h51m9.013765721s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243179531 (377h51m9.013820866s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243274663 (377h51m9.013915998s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243323757 (377h51m9.013965092s) [seg_pos={0 0}, upload=302293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 03:01:34.243412717 (377h51m9.014054052s) [upload=302293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 03:01:34.243417376 (377h51m9.014058711s) [upload=302293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 03:01:34.538397355 (377h51m9.30903869s) [upload=302293]: Done uploading segments... Jul 9 03:01:34.538417173 (377h51m9.309058508s) [upload=302293]: Upload finished. err: <nil> Duration: 5.755454774s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [.X..XXX...X.........X.XXX......X............................] duration: 5.497267712s FAIL (2025-07-09 03:20:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.497295586s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [..X..XXX...X.........X.XXX......X...........................] duration: 44.86343926s FAIL (2025-07-09 03:20:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 44.863463226s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X...X.....XXX.........X.X...................................] duration: 23.660648272s FAIL (2025-07-09 03:20:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.660684933s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.802004919s FAIL (2025-07-09 03:19:58 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== expected piece hash; context canceled; EOF Jul 9 03:00:26.821547853 (377h50m1.592189189s) [noise=true , address=45.91.116.7:4010 , piece_id=OR5D2DCY5B4IKLDVVZT73DXDUTABGQ4K4VDJEL5HBSV3SQSS4GCQ, seg_pos={0 0}, upload=302277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Jul 9 03:00:26.821590575 (377h50m1.59223191s) [seg_pos={0 0}, upload=302277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 03:00:26.821670989 (377h50m1.592312314s) [upload=302277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 03:00:26.821675889 (377h50m1.592317224s) [upload=302277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 03:00:27.129149168 (377h50m1.899790543s) [upload=302277]: Done uploading segments... Jul 9 03:00:27.129166300 (377h50m1.899807675s) [upload=302277]: Upload finished. err: <nil> Jul 9 03:01:31.849644782 (377h51m6.620286117s) [upload=302293]: Starting upload Jul 9 03:01:31.849706480 (377h51m6.620347815s) [upload=302293]: Done starting upload Jul 9 03:01:31.849908567 (377h51m6.620549902s) [upload=302293]: Uploading segments... Jul 9 03:01:31.849940148 (377h51m6.620581493s) [upload=302293]: Got next segment. Inline: false Jul 9 03:01:31.849961679 (377h51m6.620603014s) [upload=302293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 03:01:31.849966318 (377h51m6.620607663s) [upload=302293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 03:01:32.082815894 (377h51m6.853457229s) [seg_pos={0 0}, upload=302293]: Begin upload segment... Jul 9 03:01:32.082857022 (377h51m6.853498357s) [seg_pos={0 0}, upload=302293]: Done begin upload segment. Jul 9 03:01:32.082877130 (377h51m6.853518476s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.082910053 (377h51m6.853551378s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083224024 (377h51m6.853865359s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083306551 (377h51m6.853947887s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083473250 (377h51m6.854114585s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083576027 (377h51m6.854217352s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083720604 (377h51m6.854361929s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083881712 (377h51m6.854523047s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084015208 (377h51m6.854656543s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084188880 (377h51m6.854830215s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084277228 (377h51m6.854918563s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084445321 (377h51m6.855086656s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084507439 (377h51m6.855148775s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084669670 (377h51m6.855311005s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084714255 (377h51m6.85535559s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084930158 (377h51m6.855571493s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:33.718884639 (377h51m8.489525974s) [upload=302293]: Next returned nil segment Jul 9 03:01:33.718901952 (377h51m8.489543287s) [upload=302293]: Waiting for error group managing segments... Jul 9 03:01:34.050524955 (377h51m8.82116629s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.058353302 (377h51m8.828994637s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.098151442 (377h51m8.868792777s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.110943775 (377h51m8.88158511s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.117461985 (377h51m8.88810332s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.161579608 (377h51m8.932220933s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.170088417 (377h51m8.940729752s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.170743521 (377h51m8.941384846s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.176248322 (377h51m8.946889647s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.183619354 (377h51m8.954260679s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.236599983 (377h51m9.007241318s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.242137387 (377h51m9.012778712s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.242995759 (377h51m9.013637094s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.243008334 (377h51m9.013649669s) [seg_pos={0 0}, upload=302293]: Segment reached optimal threshold of 13 pieces. Jul 9 03:01:34.243124386 (377h51m9.013765721s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243179531 (377h51m9.013820866s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243274663 (377h51m9.013915998s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243323757 (377h51m9.013965092s) [seg_pos={0 0}, upload=302293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 03:01:34.243412717 (377h51m9.014054052s) [upload=302293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 03:01:34.243417376 (377h51m9.014058711s) [upload=302293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 03:01:34.538397355 (377h51m9.30903869s) [upload=302293]: Done uploading segments... Jul 9 03:01:34.538417173 (377h51m9.309058508s) [upload=302293]: Upload finished. err: <nil> Duration: 5.802035587s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X...X.....XXX.........X.X..................................] duration: 6.196292829s FAIL (2025-07-09 03:19:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.196339198s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 9.599813009s FAIL (2025-07-09 03:19:31 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== t next segment. Inline: false Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment... Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment. Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments... Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces. Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments... Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil> Duration: 9.599838809s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [...X..XXX...X.........X.XXX......X..........................] duration: 24.32769726s FAIL (2025-07-09 03:19:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.327721808s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.7364359s FAIL (2025-07-09 03:19:19 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== expected piece hash; context canceled; EOF Jul 9 03:00:26.821547853 (377h50m1.592189189s) [noise=true , address=45.91.116.7:4010 , piece_id=OR5D2DCY5B4IKLDVVZT73DXDUTABGQ4K4VDJEL5HBSV3SQSS4GCQ, seg_pos={0 0}, upload=302277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Jul 9 03:00:26.821590575 (377h50m1.59223191s) [seg_pos={0 0}, upload=302277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 03:00:26.821670989 (377h50m1.592312314s) [upload=302277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 03:00:26.821675889 (377h50m1.592317224s) [upload=302277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 03:00:27.129149168 (377h50m1.899790543s) [upload=302277]: Done uploading segments... Jul 9 03:00:27.129166300 (377h50m1.899807675s) [upload=302277]: Upload finished. err: <nil> Jul 9 03:01:31.849644782 (377h51m6.620286117s) [upload=302293]: Starting upload Jul 9 03:01:31.849706480 (377h51m6.620347815s) [upload=302293]: Done starting upload Jul 9 03:01:31.849908567 (377h51m6.620549902s) [upload=302293]: Uploading segments... Jul 9 03:01:31.849940148 (377h51m6.620581493s) [upload=302293]: Got next segment. Inline: false Jul 9 03:01:31.849961679 (377h51m6.620603014s) [upload=302293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 03:01:31.849966318 (377h51m6.620607663s) [upload=302293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 03:01:32.082815894 (377h51m6.853457229s) [seg_pos={0 0}, upload=302293]: Begin upload segment... Jul 9 03:01:32.082857022 (377h51m6.853498357s) [seg_pos={0 0}, upload=302293]: Done begin upload segment. Jul 9 03:01:32.082877130 (377h51m6.853518476s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.082910053 (377h51m6.853551378s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083224024 (377h51m6.853865359s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083306551 (377h51m6.853947887s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083473250 (377h51m6.854114585s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083576027 (377h51m6.854217352s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083720604 (377h51m6.854361929s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083881712 (377h51m6.854523047s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084015208 (377h51m6.854656543s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084188880 (377h51m6.854830215s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084277228 (377h51m6.854918563s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084445321 (377h51m6.855086656s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084507439 (377h51m6.855148775s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084669670 (377h51m6.855311005s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084714255 (377h51m6.85535559s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084930158 (377h51m6.855571493s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:33.718884639 (377h51m8.489525974s) [upload=302293]: Next returned nil segment Jul 9 03:01:33.718901952 (377h51m8.489543287s) [upload=302293]: Waiting for error group managing segments... Jul 9 03:01:34.050524955 (377h51m8.82116629s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.058353302 (377h51m8.828994637s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.098151442 (377h51m8.868792777s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.110943775 (377h51m8.88158511s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.117461985 (377h51m8.88810332s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.161579608 (377h51m8.932220933s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.170088417 (377h51m8.940729752s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.170743521 (377h51m8.941384846s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.176248322 (377h51m8.946889647s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.183619354 (377h51m8.954260679s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.236599983 (377h51m9.007241318s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.242137387 (377h51m9.012778712s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.242995759 (377h51m9.013637094s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.243008334 (377h51m9.013649669s) [seg_pos={0 0}, upload=302293]: Segment reached optimal threshold of 13 pieces. Jul 9 03:01:34.243124386 (377h51m9.013765721s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243179531 (377h51m9.013820866s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243274663 (377h51m9.013915998s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243323757 (377h51m9.013965092s) [seg_pos={0 0}, upload=302293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 03:01:34.243412717 (377h51m9.014054052s) [upload=302293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 03:01:34.243417376 (377h51m9.014058711s) [upload=302293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 03:01:34.538397355 (377h51m9.30903869s) [upload=302293]: Done uploading segments... Jul 9 03:01:34.538417173 (377h51m9.309058508s) [upload=302293]: Upload finished. err: <nil> Duration: 5.736473001s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.85985918s FAIL (2025-07-09 03:19:07 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== t next segment. Inline: false Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment... Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment. Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments... Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces. Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments... Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil> Duration: 5.859883738s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X.X...X.....XXX.........X.X.................................] duration: 6.186573802s FAIL (2025-07-09 03:18:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.186594612s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.676154649s FAIL (2025-07-09 03:18:50 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== expected piece hash; context canceled; EOF Jul 9 03:00:26.821547853 (377h50m1.592189189s) [noise=true , address=45.91.116.7:4010 , piece_id=OR5D2DCY5B4IKLDVVZT73DXDUTABGQ4K4VDJEL5HBSV3SQSS4GCQ, seg_pos={0 0}, upload=302277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Jul 9 03:00:26.821590575 (377h50m1.59223191s) [seg_pos={0 0}, upload=302277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 03:00:26.821670989 (377h50m1.592312314s) [upload=302277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 03:00:26.821675889 (377h50m1.592317224s) [upload=302277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 03:00:27.129149168 (377h50m1.899790543s) [upload=302277]: Done uploading segments... Jul 9 03:00:27.129166300 (377h50m1.899807675s) [upload=302277]: Upload finished. err: <nil> Jul 9 03:01:31.849644782 (377h51m6.620286117s) [upload=302293]: Starting upload Jul 9 03:01:31.849706480 (377h51m6.620347815s) [upload=302293]: Done starting upload Jul 9 03:01:31.849908567 (377h51m6.620549902s) [upload=302293]: Uploading segments... Jul 9 03:01:31.849940148 (377h51m6.620581493s) [upload=302293]: Got next segment. Inline: false Jul 9 03:01:31.849961679 (377h51m6.620603014s) [upload=302293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 03:01:31.849966318 (377h51m6.620607663s) [upload=302293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 03:01:32.082815894 (377h51m6.853457229s) [seg_pos={0 0}, upload=302293]: Begin upload segment... Jul 9 03:01:32.082857022 (377h51m6.853498357s) [seg_pos={0 0}, upload=302293]: Done begin upload segment. Jul 9 03:01:32.082877130 (377h51m6.853518476s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.082910053 (377h51m6.853551378s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083224024 (377h51m6.853865359s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083306551 (377h51m6.853947887s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083473250 (377h51m6.854114585s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083576027 (377h51m6.854217352s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083720604 (377h51m6.854361929s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083881712 (377h51m6.854523047s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084015208 (377h51m6.854656543s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084188880 (377h51m6.854830215s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084277228 (377h51m6.854918563s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084445321 (377h51m6.855086656s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084507439 (377h51m6.855148775s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084669670 (377h51m6.855311005s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084714255 (377h51m6.85535559s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084930158 (377h51m6.855571493s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:33.718884639 (377h51m8.489525974s) [upload=302293]: Next returned nil segment Jul 9 03:01:33.718901952 (377h51m8.489543287s) [upload=302293]: Waiting for error group managing segments... Jul 9 03:01:34.050524955 (377h51m8.82116629s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.058353302 (377h51m8.828994637s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.098151442 (377h51m8.868792777s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.110943775 (377h51m8.88158511s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.117461985 (377h51m8.88810332s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.161579608 (377h51m8.932220933s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.170088417 (377h51m8.940729752s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.170743521 (377h51m8.941384846s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.176248322 (377h51m8.946889647s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.183619354 (377h51m8.954260679s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.236599983 (377h51m9.007241318s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.242137387 (377h51m9.012778712s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.242995759 (377h51m9.013637094s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.243008334 (377h51m9.013649669s) [seg_pos={0 0}, upload=302293]: Segment reached optimal threshold of 13 pieces. Jul 9 03:01:34.243124386 (377h51m9.013765721s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243179531 (377h51m9.013820866s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243274663 (377h51m9.013915998s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243323757 (377h51m9.013965092s) [seg_pos={0 0}, upload=302293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 03:01:34.243412717 (377h51m9.014054052s) [upload=302293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 03:01:34.243417376 (377h51m9.014058711s) [upload=302293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 03:01:34.538397355 (377h51m9.30903869s) [upload=302293]: Done uploading segments... Jul 9 03:01:34.538417173 (377h51m9.309058508s) [upload=302293]: Upload finished. err: <nil> Duration: 5.676176151s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [XX.X...X.....XXX.........X.X................................] duration: 6.317876149s FAIL (2025-07-09 03:18:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.317894414s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [....X..XXX...X.........X.XXX......X.........................] duration: 5.500585408s FAIL (2025-07-09 03:18:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.500610116s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.805026681s FAIL (2025-07-09 03:18:14 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== t next segment. Inline: false Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment... Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment. Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments... Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces. Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments... Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil> Duration: 5.80505229s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.XX.X...X.....XXX.........X.X...............................] duration: 6.243474305s FAIL (2025-07-09 03:18:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.243493071s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.69042579s FAIL (2025-07-09 03:17:55 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== expected piece hash; context canceled; EOF Jul 9 03:00:26.821547853 (377h50m1.592189189s) [noise=true , address=45.91.116.7:4010 , piece_id=OR5D2DCY5B4IKLDVVZT73DXDUTABGQ4K4VDJEL5HBSV3SQSS4GCQ, seg_pos={0 0}, upload=302277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Jul 9 03:00:26.821590575 (377h50m1.59223191s) [seg_pos={0 0}, upload=302277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 03:00:26.821670989 (377h50m1.592312314s) [upload=302277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 03:00:26.821675889 (377h50m1.592317224s) [upload=302277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 03:00:27.129149168 (377h50m1.899790543s) [upload=302277]: Done uploading segments... Jul 9 03:00:27.129166300 (377h50m1.899807675s) [upload=302277]: Upload finished. err: <nil> Jul 9 03:01:31.849644782 (377h51m6.620286117s) [upload=302293]: Starting upload Jul 9 03:01:31.849706480 (377h51m6.620347815s) [upload=302293]: Done starting upload Jul 9 03:01:31.849908567 (377h51m6.620549902s) [upload=302293]: Uploading segments... Jul 9 03:01:31.849940148 (377h51m6.620581493s) [upload=302293]: Got next segment. Inline: false Jul 9 03:01:31.849961679 (377h51m6.620603014s) [upload=302293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 03:01:31.849966318 (377h51m6.620607663s) [upload=302293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 03:01:32.082815894 (377h51m6.853457229s) [seg_pos={0 0}, upload=302293]: Begin upload segment... Jul 9 03:01:32.082857022 (377h51m6.853498357s) [seg_pos={0 0}, upload=302293]: Done begin upload segment. Jul 9 03:01:32.082877130 (377h51m6.853518476s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.082910053 (377h51m6.853551378s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083224024 (377h51m6.853865359s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083306551 (377h51m6.853947887s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083473250 (377h51m6.854114585s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083576027 (377h51m6.854217352s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083720604 (377h51m6.854361929s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083881712 (377h51m6.854523047s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084015208 (377h51m6.854656543s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084188880 (377h51m6.854830215s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084277228 (377h51m6.854918563s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084445321 (377h51m6.855086656s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084507439 (377h51m6.855148775s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084669670 (377h51m6.855311005s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084714255 (377h51m6.85535559s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084930158 (377h51m6.855571493s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:33.718884639 (377h51m8.489525974s) [upload=302293]: Next returned nil segment Jul 9 03:01:33.718901952 (377h51m8.489543287s) [upload=302293]: Waiting for error group managing segments... Jul 9 03:01:34.050524955 (377h51m8.82116629s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.058353302 (377h51m8.828994637s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.098151442 (377h51m8.868792777s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.110943775 (377h51m8.88158511s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.117461985 (377h51m8.88810332s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.161579608 (377h51m8.932220933s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.170088417 (377h51m8.940729752s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.170743521 (377h51m8.941384846s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.176248322 (377h51m8.946889647s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.183619354 (377h51m8.954260679s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.236599983 (377h51m9.007241318s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.242137387 (377h51m9.012778712s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.242995759 (377h51m9.013637094s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.243008334 (377h51m9.013649669s) [seg_pos={0 0}, upload=302293]: Segment reached optimal threshold of 13 pieces. Jul 9 03:01:34.243124386 (377h51m9.013765721s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243179531 (377h51m9.013820866s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243274663 (377h51m9.013915998s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243323757 (377h51m9.013965092s) [seg_pos={0 0}, upload=302293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 03:01:34.243412717 (377h51m9.014054052s) [upload=302293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 03:01:34.243417376 (377h51m9.014058711s) [upload=302293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 03:01:34.538397355 (377h51m9.30903869s) [upload=302293]: Done uploading segments... Jul 9 03:01:34.538417173 (377h51m9.309058508s) [upload=302293]: Upload finished. err: <nil> Duration: 5.690445788s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.827413759s FAIL (2025-07-09 03:17:15 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== t next segment. Inline: false Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment... Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment. Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments... Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces. Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments... Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil> Duration: 5.827432274s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [X....X..XXX...X.........X.XXX......X........................] duration: 5.687684437s FAIL (2025-07-09 03:17:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.687705638s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.803918077s FAIL (2025-07-09 03:17:09 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== expected piece hash; context canceled; EOF Jul 9 03:00:26.821547853 (377h50m1.592189189s) [noise=true , address=45.91.116.7:4010 , piece_id=OR5D2DCY5B4IKLDVVZT73DXDUTABGQ4K4VDJEL5HBSV3SQSS4GCQ, seg_pos={0 0}, upload=302277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Jul 9 03:00:26.821590575 (377h50m1.59223191s) [seg_pos={0 0}, upload=302277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 03:00:26.821670989 (377h50m1.592312314s) [upload=302277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 03:00:26.821675889 (377h50m1.592317224s) [upload=302277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 03:00:27.129149168 (377h50m1.899790543s) [upload=302277]: Done uploading segments... Jul 9 03:00:27.129166300 (377h50m1.899807675s) [upload=302277]: Upload finished. err: <nil> Jul 9 03:01:31.849644782 (377h51m6.620286117s) [upload=302293]: Starting upload Jul 9 03:01:31.849706480 (377h51m6.620347815s) [upload=302293]: Done starting upload Jul 9 03:01:31.849908567 (377h51m6.620549902s) [upload=302293]: Uploading segments... Jul 9 03:01:31.849940148 (377h51m6.620581493s) [upload=302293]: Got next segment. Inline: false Jul 9 03:01:31.849961679 (377h51m6.620603014s) [upload=302293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 03:01:31.849966318 (377h51m6.620607663s) [upload=302293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 03:01:32.082815894 (377h51m6.853457229s) [seg_pos={0 0}, upload=302293]: Begin upload segment... Jul 9 03:01:32.082857022 (377h51m6.853498357s) [seg_pos={0 0}, upload=302293]: Done begin upload segment. Jul 9 03:01:32.082877130 (377h51m6.853518476s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.082910053 (377h51m6.853551378s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083224024 (377h51m6.853865359s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083306551 (377h51m6.853947887s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083473250 (377h51m6.854114585s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083576027 (377h51m6.854217352s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083720604 (377h51m6.854361929s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083881712 (377h51m6.854523047s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084015208 (377h51m6.854656543s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084188880 (377h51m6.854830215s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084277228 (377h51m6.854918563s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084445321 (377h51m6.855086656s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084507439 (377h51m6.855148775s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084669670 (377h51m6.855311005s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084714255 (377h51m6.85535559s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084930158 (377h51m6.855571493s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:33.718884639 (377h51m8.489525974s) [upload=302293]: Next returned nil segment Jul 9 03:01:33.718901952 (377h51m8.489543287s) [upload=302293]: Waiting for error group managing segments... Jul 9 03:01:34.050524955 (377h51m8.82116629s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.058353302 (377h51m8.828994637s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.098151442 (377h51m8.868792777s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.110943775 (377h51m8.88158511s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.117461985 (377h51m8.88810332s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.161579608 (377h51m8.932220933s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.170088417 (377h51m8.940729752s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.170743521 (377h51m8.941384846s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.176248322 (377h51m8.946889647s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.183619354 (377h51m8.954260679s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.236599983 (377h51m9.007241318s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.242137387 (377h51m9.012778712s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.242995759 (377h51m9.013637094s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.243008334 (377h51m9.013649669s) [seg_pos={0 0}, upload=302293]: Segment reached optimal threshold of 13 pieces. Jul 9 03:01:34.243124386 (377h51m9.013765721s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243179531 (377h51m9.013820866s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243274663 (377h51m9.013915998s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243323757 (377h51m9.013965092s) [seg_pos={0 0}, upload=302293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 03:01:34.243412717 (377h51m9.014054052s) [upload=302293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 03:01:34.243417376 (377h51m9.014058711s) [upload=302293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 03:01:34.538397355 (377h51m9.30903869s) [upload=302293]: Done uploading segments... Jul 9 03:01:34.538417173 (377h51m9.309058508s) [upload=302293]: Upload finished. err: <nil> Duration: 5.803944248s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..XX.X...X.....XXX.........X.X..............................] duration: 6.310231038s FAIL (2025-07-09 03:17:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.310251526s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.756016346s FAIL (2025-07-09 03:16:49 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== t next segment. Inline: false Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment... Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment. Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments... Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces. Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments... Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil> Duration: 5.756077212s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [.X....X..XXX...X.........X.XXX......X.......................] duration: 5.506538616s FAIL (2025-07-09 03:16:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.506558725s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...XX.X...X.....XXX.........X.X.............................] duration: 5.890290314s FAIL (2025-07-09 03:16:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.890312906s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [....XX.X...X.....XXX.........X.X............................] duration: 6.173353304s FAIL (2025-07-09 03:16:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.173379045s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.974342264s FAIL (2025-07-09 03:16:02 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== t next segment. Inline: false Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment... Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment. Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments... Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces. Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments... Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil> Duration: 5.974367312s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.784780136s FAIL (2025-07-09 03:15:55 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== expected piece hash; context canceled; EOF Jul 9 03:00:26.821547853 (377h50m1.592189189s) [noise=true , address=45.91.116.7:4010 , piece_id=OR5D2DCY5B4IKLDVVZT73DXDUTABGQ4K4VDJEL5HBSV3SQSS4GCQ, seg_pos={0 0}, upload=302277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Jul 9 03:00:26.821590575 (377h50m1.59223191s) [seg_pos={0 0}, upload=302277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 03:00:26.821670989 (377h50m1.592312314s) [upload=302277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 03:00:26.821675889 (377h50m1.592317224s) [upload=302277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 03:00:27.129149168 (377h50m1.899790543s) [upload=302277]: Done uploading segments... Jul 9 03:00:27.129166300 (377h50m1.899807675s) [upload=302277]: Upload finished. err: <nil> Jul 9 03:01:31.849644782 (377h51m6.620286117s) [upload=302293]: Starting upload Jul 9 03:01:31.849706480 (377h51m6.620347815s) [upload=302293]: Done starting upload Jul 9 03:01:31.849908567 (377h51m6.620549902s) [upload=302293]: Uploading segments... Jul 9 03:01:31.849940148 (377h51m6.620581493s) [upload=302293]: Got next segment. Inline: false Jul 9 03:01:31.849961679 (377h51m6.620603014s) [upload=302293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 03:01:31.849966318 (377h51m6.620607663s) [upload=302293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 03:01:32.082815894 (377h51m6.853457229s) [seg_pos={0 0}, upload=302293]: Begin upload segment... Jul 9 03:01:32.082857022 (377h51m6.853498357s) [seg_pos={0 0}, upload=302293]: Done begin upload segment. Jul 9 03:01:32.082877130 (377h51m6.853518476s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.082910053 (377h51m6.853551378s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083224024 (377h51m6.853865359s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083306551 (377h51m6.853947887s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083473250 (377h51m6.854114585s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083576027 (377h51m6.854217352s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083720604 (377h51m6.854361929s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083881712 (377h51m6.854523047s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084015208 (377h51m6.854656543s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084188880 (377h51m6.854830215s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084277228 (377h51m6.854918563s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084445321 (377h51m6.855086656s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084507439 (377h51m6.855148775s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084669670 (377h51m6.855311005s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084714255 (377h51m6.85535559s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084930158 (377h51m6.855571493s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:33.718884639 (377h51m8.489525974s) [upload=302293]: Next returned nil segment Jul 9 03:01:33.718901952 (377h51m8.489543287s) [upload=302293]: Waiting for error group managing segments... Jul 9 03:01:34.050524955 (377h51m8.82116629s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.058353302 (377h51m8.828994637s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.098151442 (377h51m8.868792777s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.110943775 (377h51m8.88158511s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.117461985 (377h51m8.88810332s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.161579608 (377h51m8.932220933s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.170088417 (377h51m8.940729752s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.170743521 (377h51m8.941384846s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.176248322 (377h51m8.946889647s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.183619354 (377h51m8.954260679s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.236599983 (377h51m9.007241318s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.242137387 (377h51m9.012778712s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.242995759 (377h51m9.013637094s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.243008334 (377h51m9.013649669s) [seg_pos={0 0}, upload=302293]: Segment reached optimal threshold of 13 pieces. Jul 9 03:01:34.243124386 (377h51m9.013765721s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243179531 (377h51m9.013820866s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243274663 (377h51m9.013915998s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243323757 (377h51m9.013965092s) [seg_pos={0 0}, upload=302293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 03:01:34.243412717 (377h51m9.014054052s) [upload=302293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 03:01:34.243417376 (377h51m9.014058711s) [upload=302293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 03:01:34.538397355 (377h51m9.30903869s) [upload=302293]: Done uploading segments... Jul 9 03:01:34.538417173 (377h51m9.309058508s) [upload=302293]: Upload finished. err: <nil> Duration: 5.784808959s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [..X....X..XXX...X.........X.XXX......X......................] duration: 5.466201335s FAIL (2025-07-09 03:15:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.466228337s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.....XX.X...X.....XXX.........X.X...........................] duration: 6.126681732s FAIL (2025-07-09 03:15:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.126704035s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [...X....X..XXX...X.........X.XXX......X.....................] duration: 5.574832712s FAIL (2025-07-09 03:15:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.57485804s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.773295782s FAIL (2025-07-09 03:15:19 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== expected piece hash; context canceled; EOF Jul 9 03:00:26.821547853 (377h50m1.592189189s) [noise=true , address=45.91.116.7:4010 , piece_id=OR5D2DCY5B4IKLDVVZT73DXDUTABGQ4K4VDJEL5HBSV3SQSS4GCQ, seg_pos={0 0}, upload=302277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Jul 9 03:00:26.821590575 (377h50m1.59223191s) [seg_pos={0 0}, upload=302277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 03:00:26.821670989 (377h50m1.592312314s) [upload=302277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 03:00:26.821675889 (377h50m1.592317224s) [upload=302277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 03:00:27.129149168 (377h50m1.899790543s) [upload=302277]: Done uploading segments... Jul 9 03:00:27.129166300 (377h50m1.899807675s) [upload=302277]: Upload finished. err: <nil> Jul 9 03:01:31.849644782 (377h51m6.620286117s) [upload=302293]: Starting upload Jul 9 03:01:31.849706480 (377h51m6.620347815s) [upload=302293]: Done starting upload Jul 9 03:01:31.849908567 (377h51m6.620549902s) [upload=302293]: Uploading segments... Jul 9 03:01:31.849940148 (377h51m6.620581493s) [upload=302293]: Got next segment. Inline: false Jul 9 03:01:31.849961679 (377h51m6.620603014s) [upload=302293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 03:01:31.849966318 (377h51m6.620607663s) [upload=302293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 03:01:32.082815894 (377h51m6.853457229s) [seg_pos={0 0}, upload=302293]: Begin upload segment... Jul 9 03:01:32.082857022 (377h51m6.853498357s) [seg_pos={0 0}, upload=302293]: Done begin upload segment. Jul 9 03:01:32.082877130 (377h51m6.853518476s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.082910053 (377h51m6.853551378s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083224024 (377h51m6.853865359s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083306551 (377h51m6.853947887s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083473250 (377h51m6.854114585s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083576027 (377h51m6.854217352s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083720604 (377h51m6.854361929s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083881712 (377h51m6.854523047s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084015208 (377h51m6.854656543s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084188880 (377h51m6.854830215s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084277228 (377h51m6.854918563s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084445321 (377h51m6.855086656s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084507439 (377h51m6.855148775s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084669670 (377h51m6.855311005s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084714255 (377h51m6.85535559s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084930158 (377h51m6.855571493s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:33.718884639 (377h51m8.489525974s) [upload=302293]: Next returned nil segment Jul 9 03:01:33.718901952 (377h51m8.489543287s) [upload=302293]: Waiting for error group managing segments... Jul 9 03:01:34.050524955 (377h51m8.82116629s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.058353302 (377h51m8.828994637s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.098151442 (377h51m8.868792777s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.110943775 (377h51m8.88158511s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.117461985 (377h51m8.88810332s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.161579608 (377h51m8.932220933s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.170088417 (377h51m8.940729752s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.170743521 (377h51m8.941384846s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.176248322 (377h51m8.946889647s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.183619354 (377h51m8.954260679s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.236599983 (377h51m9.007241318s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.242137387 (377h51m9.012778712s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.242995759 (377h51m9.013637094s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.243008334 (377h51m9.013649669s) [seg_pos={0 0}, upload=302293]: Segment reached optimal threshold of 13 pieces. Jul 9 03:01:34.243124386 (377h51m9.013765721s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243179531 (377h51m9.013820866s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243274663 (377h51m9.013915998s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243323757 (377h51m9.013965092s) [seg_pos={0 0}, upload=302293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 03:01:34.243412717 (377h51m9.014054052s) [upload=302293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 03:01:34.243417376 (377h51m9.014058711s) [upload=302293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 03:01:34.538397355 (377h51m9.30903869s) [upload=302293]: Done uploading segments... Jul 9 03:01:34.538417173 (377h51m9.309058508s) [upload=302293]: Upload finished. err: <nil> Duration: 5.773326992s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.840730504s FAIL (2025-07-09 03:15:02 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== t next segment. Inline: false Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment... Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment. Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments... Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces. Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments... Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil> Duration: 5.840829434s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [X...X....X..XXX...X.........X.XXX......X....................] duration: 5.561315927s FAIL (2025-07-09 03:15:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.561345995s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [X.....XX.X...X.....XXX.........X.X..........................] duration: 6.111848384s FAIL (2025-07-09 03:14:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.111862453s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.724411674s FAIL (2025-07-09 03:14:10 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== expected piece hash; context canceled; EOF Jul 9 03:00:26.821547853 (377h50m1.592189189s) [noise=true , address=45.91.116.7:4010 , piece_id=OR5D2DCY5B4IKLDVVZT73DXDUTABGQ4K4VDJEL5HBSV3SQSS4GCQ, seg_pos={0 0}, upload=302277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Jul 9 03:00:26.821590575 (377h50m1.59223191s) [seg_pos={0 0}, upload=302277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 03:00:26.821670989 (377h50m1.592312314s) [upload=302277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 03:00:26.821675889 (377h50m1.592317224s) [upload=302277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 03:00:27.129149168 (377h50m1.899790543s) [upload=302277]: Done uploading segments... Jul 9 03:00:27.129166300 (377h50m1.899807675s) [upload=302277]: Upload finished. err: <nil> Jul 9 03:01:31.849644782 (377h51m6.620286117s) [upload=302293]: Starting upload Jul 9 03:01:31.849706480 (377h51m6.620347815s) [upload=302293]: Done starting upload Jul 9 03:01:31.849908567 (377h51m6.620549902s) [upload=302293]: Uploading segments... Jul 9 03:01:31.849940148 (377h51m6.620581493s) [upload=302293]: Got next segment. Inline: false Jul 9 03:01:31.849961679 (377h51m6.620603014s) [upload=302293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 03:01:31.849966318 (377h51m6.620607663s) [upload=302293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 03:01:32.082815894 (377h51m6.853457229s) [seg_pos={0 0}, upload=302293]: Begin upload segment... Jul 9 03:01:32.082857022 (377h51m6.853498357s) [seg_pos={0 0}, upload=302293]: Done begin upload segment. Jul 9 03:01:32.082877130 (377h51m6.853518476s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.082910053 (377h51m6.853551378s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083224024 (377h51m6.853865359s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083306551 (377h51m6.853947887s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083473250 (377h51m6.854114585s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083576027 (377h51m6.854217352s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083720604 (377h51m6.854361929s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083881712 (377h51m6.854523047s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084015208 (377h51m6.854656543s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084188880 (377h51m6.854830215s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084277228 (377h51m6.854918563s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084445321 (377h51m6.855086656s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084507439 (377h51m6.855148775s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084669670 (377h51m6.855311005s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084714255 (377h51m6.85535559s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084930158 (377h51m6.855571493s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:33.718884639 (377h51m8.489525974s) [upload=302293]: Next returned nil segment Jul 9 03:01:33.718901952 (377h51m8.489543287s) [upload=302293]: Waiting for error group managing segments... Jul 9 03:01:34.050524955 (377h51m8.82116629s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.058353302 (377h51m8.828994637s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.098151442 (377h51m8.868792777s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.110943775 (377h51m8.88158511s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.117461985 (377h51m8.88810332s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.161579608 (377h51m8.932220933s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.170088417 (377h51m8.940729752s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.170743521 (377h51m8.941384846s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.176248322 (377h51m8.946889647s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.183619354 (377h51m8.954260679s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.236599983 (377h51m9.007241318s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.242137387 (377h51m9.012778712s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.242995759 (377h51m9.013637094s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.243008334 (377h51m9.013649669s) [seg_pos={0 0}, upload=302293]: Segment reached optimal threshold of 13 pieces. Jul 9 03:01:34.243124386 (377h51m9.013765721s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243179531 (377h51m9.013820866s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243274663 (377h51m9.013915998s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243323757 (377h51m9.013965092s) [seg_pos={0 0}, upload=302293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 03:01:34.243412717 (377h51m9.014054052s) [upload=302293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 03:01:34.243417376 (377h51m9.014058711s) [upload=302293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 03:01:34.538397355 (377h51m9.30903869s) [upload=302293]: Done uploading segments... Jul 9 03:01:34.538417173 (377h51m9.309058508s) [upload=302293]: Upload finished. err: <nil> Duration: 5.724436261s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.906132359s FAIL (2025-07-09 03:13:55 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== t next segment. Inline: false Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment... Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment. Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments... Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces. Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments... Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil> Duration: 5.906158989s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [XX...X....X..XXX...X.........X.XXX......X...................] duration: 5.531852856s FAIL (2025-07-09 03:13:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.531872785s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [XX.....XX.X...X.....XXX.........X.X.........................] duration: 6.165580661s FAIL (2025-07-09 03:13:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.165596702s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.624155899s FAIL (2025-07-09 03:13:38 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== t next segment. Inline: false Jul 9 02:34:19.356840408 (377h23m54.127481753s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 02:34:19.356849466 (377h23m54.127490801s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 02:34:19.798061466 (377h23m54.568702801s) [seg_pos={0 0}, upload=301931]: Begin upload segment... Jul 9 02:34:19.798104257 (377h23m54.568745592s) [seg_pos={0 0}, upload=301931]: Done begin upload segment. Jul 9 02:34:19.798125357 (377h23m54.568766693s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798144104 (377h23m54.568785439s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798487200 (377h23m54.569128535s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798498101 (377h23m54.569139436s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798739693 (377h23m54.569381028s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.798756675 (377h23m54.56939801s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799051780 (377h23m54.569693145s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799065987 (377h23m54.569707312s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799314773 (377h23m54.569956108s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799338258 (377h23m54.569979593s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799522711 (377h23m54.570164046s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.799748513 (377h23m54.570389838s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800064597 (377h23m54.570705922s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.800731544 (377h23m54.571372879s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801002311 (377h23m54.571643646s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801104958 (377h23m54.571746293s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.801943833 (377h23m54.572585169s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:19.802343307 (377h23m54.572984642s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Uploading piece... Jul 9 02:34:21.314256952 (377h23m56.084898287s) [upload=301931]: Next returned nil segment Jul 9 02:34:21.314275226 (377h23m56.084916561s) [upload=301931]: Waiting for error group managing segments... Jul 9 02:34:21.733989859 (377h23m56.504631194s) [noise=true , address=156.146.43.225:4468 , piece_id=P4HUI4A374GLIW43FUXFFBGF6DSX2ZBFECDPBC26CO65ZWRTFBBQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.762146136 (377h23m56.532787471s) [noise=true , address=45.91.116.5:4331 , piece_id=EM6ELNAZZK7ED2PQV47RMTE4VBWM3HQBHLDUAFXHGL7D4HPINU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.790197392 (377h23m56.560838727s) [noise=true , address=77.90.31.12:4594 , piece_id=CVTOQ6TZRUMGJJFGEYL3AKZDKYWRWPSKOJADUL347PQJUP5RLACQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.803954852 (377h23m56.574596187s) [noise=true , address=77.90.31.14:4679 , piece_id=HKQSS3GEPNSAJYUUMS6BNVOHFFYUW5IRNVF5UQPHKB42PYUHJINQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.804471321 (377h23m56.575112656s) [noise=true , address=77.90.31.19:4852 , piece_id=SJHKXCTU6FAVUNATDB2AR2RLLCJTWDAZ3BJLC5RKDW5TKD5NEVHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.806459134 (377h23m56.577100459s) [noise=true , address=77.90.31.13:4656 , piece_id=6UXUHDO65D4ZHL5NARNBULFKMWK2X36I32PVTXE7UPBAPTT5TYDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.808961061 (377h23m56.579602396s) [noise=true , address=77.90.31.17:4773 , piece_id=Y73GUKHUMVWY3JOJQVZ6LKVETWULHOAPVX7SRWMDLCQVHL6S5UHA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.818310757 (377h23m56.588952092s) [noise=true , address=77.90.31.16:4756 , piece_id=PDPJQSGU2PRQJUPNQ42X7LA4LEDXKM2YODMAUVU4R63MQWTVMOXQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:21.833220272 (377h23m56.603861607s) [noise=true , address=77.90.31.15:4722 , piece_id=7WBO6L55E42U5GM2UYFVIUTED3Y6BBYADL3PONQOOPJLE3I5FN2A, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:34:34.871763084 (377h24m9.642404419s) [noise=true , address=104.36.82.60:4355 , piece_id=3JXDKE6GN7IOXFW67VFMM6E5OZPAAFU2B6JSX6FBRMA33V45EEUA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:00.641279658 (377h24m35.411920993s) [noise=true , address=104.36.82.44:4882 , piece_id=WMUW3WS5TMWVJSZ7QFTDP263EWAX62YCOSO6N6PHJCP6WS5RIU3Q, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:01.033862472 (377h24m35.804503807s) [noise=true , address=104.36.82.21:4506 , piece_id=NKGEVSZHG33QHVJD7DOQUB2C2L25XOI6YPSB4KT262FGBDKA2CVQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:11.485065354 (377h24m46.255706679s) [noise=true , address=104.36.82.22:4512 , piece_id=JLDZMG3QZFNDVAJP6VTPFSCHLMLPAP55QWHSV6PJQISUMWQMCGKA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731152434 (377h24m51.501793769s) [noise=true , address=104.36.82.12:4181 , piece_id=G2B2PQYOQY27ZUHTBPUZCXW5JNN7SI3FOWSDQ7W32ZNED6VUZJDA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: <nil> Jul 9 02:35:16.731178774 (377h24m51.50182011s) [seg_pos={0 0}, upload=301931]: Segment reached optimal threshold of 14 pieces. Jul 9 02:35:16.731274327 (377h24m51.501915662s) [noise=true , address=104.36.82.23:4477 , piece_id=KG3WCKK2XS7LRFLQW3P3T74M5YML5IVW6X3IJFL5KOA2HPAYG6GA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12RP2Zq643WWNtKEr34Tq1iZS29NYA2okr9Fg1jiCpaFTqfLauf): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731306127 (377h24m51.501947463s) [noise=true , address=104.36.82.36:4642 , piece_id=BMBWKBPXYBDHMXATPDIUXJALEVGNUL7R5YIWGDAHSO2OL6LNVZQQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127tVQUAwStYLonZ1jjq2tjtuqbLpKh593Uj1yyhGPy5XQ5AF5L): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731350363 (377h24m51.501991698s) [noise=true , address=104.36.82.40:4963 , piece_id=TJSJEC7JTU5JR6WHAXWXINE6X2FXR4IGBIO7A3JNLT4KEJCNZOOQ, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KBrPwyuwtjG5mUijCAbHGGRZAKTBMHJBYsXLmfGx9mKRedLwm): protocol: expected piece hash; context canceled; EOF Jul 9 02:35:16.731398695 (377h24m51.50204003s) [noise=true , address=104.36.82.17:4342 , piece_id=LHZGS7L2AS5AP6PDZQRL63XQI4ZF3K6VJ6IAF6VERSDADK5EGLCA, seg_pos={0 0}, upload=301931]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBkYmDgrDUtt5dy7neY9766k7QmihsneA8JxwksWWDMGK5t9P): protocol: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34370->104.36.82.17:4342: use of closed network connection Jul 9 02:35:16.731417731 (377h24m51.502059066s) [seg_pos={0 0}, upload=301931]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 02:35:16.731479168 (377h24m51.502120503s) [upload=301931]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 02:35:16.731485631 (377h24m51.502126966s) [upload=301931]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 02:35:17.055624482 (377h24m51.826265817s) [upload=301931]: Done uploading segments... Jul 9 02:35:17.055644691 (377h24m51.826286026s) [upload=301931]: Upload finished. err: <nil> Duration: 5.624182098s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.810599962s FAIL (2025-07-09 03:13:34 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== expected piece hash; context canceled; EOF Jul 9 03:00:26.821547853 (377h50m1.592189189s) [noise=true , address=45.91.116.7:4010 , piece_id=OR5D2DCY5B4IKLDVVZT73DXDUTABGQ4K4VDJEL5HBSV3SQSS4GCQ, seg_pos={0 0}, upload=302277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Jul 9 03:00:26.821590575 (377h50m1.59223191s) [seg_pos={0 0}, upload=302277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 03:00:26.821670989 (377h50m1.592312314s) [upload=302277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 03:00:26.821675889 (377h50m1.592317224s) [upload=302277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 03:00:27.129149168 (377h50m1.899790543s) [upload=302277]: Done uploading segments... Jul 9 03:00:27.129166300 (377h50m1.899807675s) [upload=302277]: Upload finished. err: <nil> Jul 9 03:01:31.849644782 (377h51m6.620286117s) [upload=302293]: Starting upload Jul 9 03:01:31.849706480 (377h51m6.620347815s) [upload=302293]: Done starting upload Jul 9 03:01:31.849908567 (377h51m6.620549902s) [upload=302293]: Uploading segments... Jul 9 03:01:31.849940148 (377h51m6.620581493s) [upload=302293]: Got next segment. Inline: false Jul 9 03:01:31.849961679 (377h51m6.620603014s) [upload=302293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 03:01:31.849966318 (377h51m6.620607663s) [upload=302293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 03:01:32.082815894 (377h51m6.853457229s) [seg_pos={0 0}, upload=302293]: Begin upload segment... Jul 9 03:01:32.082857022 (377h51m6.853498357s) [seg_pos={0 0}, upload=302293]: Done begin upload segment. Jul 9 03:01:32.082877130 (377h51m6.853518476s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.082910053 (377h51m6.853551378s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083224024 (377h51m6.853865359s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083306551 (377h51m6.853947887s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083473250 (377h51m6.854114585s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083576027 (377h51m6.854217352s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083720604 (377h51m6.854361929s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083881712 (377h51m6.854523047s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084015208 (377h51m6.854656543s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084188880 (377h51m6.854830215s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084277228 (377h51m6.854918563s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084445321 (377h51m6.855086656s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084507439 (377h51m6.855148775s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084669670 (377h51m6.855311005s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084714255 (377h51m6.85535559s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084930158 (377h51m6.855571493s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:33.718884639 (377h51m8.489525974s) [upload=302293]: Next returned nil segment Jul 9 03:01:33.718901952 (377h51m8.489543287s) [upload=302293]: Waiting for error group managing segments... Jul 9 03:01:34.050524955 (377h51m8.82116629s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.058353302 (377h51m8.828994637s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.098151442 (377h51m8.868792777s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.110943775 (377h51m8.88158511s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.117461985 (377h51m8.88810332s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.161579608 (377h51m8.932220933s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.170088417 (377h51m8.940729752s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.170743521 (377h51m8.941384846s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.176248322 (377h51m8.946889647s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.183619354 (377h51m8.954260679s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.236599983 (377h51m9.007241318s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.242137387 (377h51m9.012778712s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.242995759 (377h51m9.013637094s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.243008334 (377h51m9.013649669s) [seg_pos={0 0}, upload=302293]: Segment reached optimal threshold of 13 pieces. Jul 9 03:01:34.243124386 (377h51m9.013765721s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243179531 (377h51m9.013820866s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243274663 (377h51m9.013915998s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243323757 (377h51m9.013965092s) [seg_pos={0 0}, upload=302293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 03:01:34.243412717 (377h51m9.014054052s) [upload=302293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 03:01:34.243417376 (377h51m9.014058711s) [upload=302293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 03:01:34.538397355 (377h51m9.30903869s) [upload=302293]: Done uploading segments... Jul 9 03:01:34.538417173 (377h51m9.309058508s) [upload=302293]: Upload finished. err: <nil> Duration: 5.810650118s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.XX.....XX.X...X.....XXX.........X.X........................] duration: 5.885329723s FAIL (2025-07-09 03:13:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.885350281s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.74508043s FAIL (2025-07-09 03:13:19 UTC) orbiter warning: ======ERROR====== uplink: bucket: metaclient: context deadline exceeded ======LOGS====== expected piece hash; context canceled; EOF Jul 9 03:00:26.821547853 (377h50m1.592189189s) [noise=true , address=45.91.116.7:4010 , piece_id=OR5D2DCY5B4IKLDVVZT73DXDUTABGQ4K4VDJEL5HBSV3SQSS4GCQ, seg_pos={0 0}, upload=302277]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Jul 9 03:00:26.821590575 (377h50m1.59223191s) [seg_pos={0 0}, upload=302277]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 03:00:26.821670989 (377h50m1.592312314s) [upload=302277]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 03:00:26.821675889 (377h50m1.592317224s) [upload=302277]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 03:00:27.129149168 (377h50m1.899790543s) [upload=302277]: Done uploading segments... Jul 9 03:00:27.129166300 (377h50m1.899807675s) [upload=302277]: Upload finished. err: <nil> Jul 9 03:01:31.849644782 (377h51m6.620286117s) [upload=302293]: Starting upload Jul 9 03:01:31.849706480 (377h51m6.620347815s) [upload=302293]: Done starting upload Jul 9 03:01:31.849908567 (377h51m6.620549902s) [upload=302293]: Uploading segments... Jul 9 03:01:31.849940148 (377h51m6.620581493s) [upload=302293]: Got next segment. Inline: false Jul 9 03:01:31.849961679 (377h51m6.620603014s) [upload=302293]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 03:01:31.849966318 (377h51m6.620607663s) [upload=302293]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 03:01:32.082815894 (377h51m6.853457229s) [seg_pos={0 0}, upload=302293]: Begin upload segment... Jul 9 03:01:32.082857022 (377h51m6.853498357s) [seg_pos={0 0}, upload=302293]: Done begin upload segment. Jul 9 03:01:32.082877130 (377h51m6.853518476s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.082910053 (377h51m6.853551378s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083224024 (377h51m6.853865359s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083306551 (377h51m6.853947887s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083473250 (377h51m6.854114585s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083576027 (377h51m6.854217352s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083720604 (377h51m6.854361929s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.083881712 (377h51m6.854523047s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084015208 (377h51m6.854656543s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084188880 (377h51m6.854830215s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084277228 (377h51m6.854918563s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084445321 (377h51m6.855086656s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084507439 (377h51m6.855148775s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084669670 (377h51m6.855311005s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084714255 (377h51m6.85535559s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:32.084930158 (377h51m6.855571493s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Uploading piece... Jul 9 03:01:33.718884639 (377h51m8.489525974s) [upload=302293]: Next returned nil segment Jul 9 03:01:33.718901952 (377h51m8.489543287s) [upload=302293]: Waiting for error group managing segments... Jul 9 03:01:34.050524955 (377h51m8.82116629s) [noise=true , address=104.36.82.11:4113 , piece_id=GPP2TC5NDK6GFNS4RHUUVQBKRLOLOBRQ2ZOYBWYB75ANW2EDGRRA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.058353302 (377h51m8.828994637s) [noise=true , address=104.36.82.2:4730 , piece_id=GSGFMMORJB3LEGHOBI7AHOOT64SLKLC3IBMMKGK4ZHET4APJ3HKA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.098151442 (377h51m8.868792777s) [noise=true , address=104.36.82.16:4295 , piece_id=BTGO6HOLM4QZT4ORXAYNDVXVVJY365HIHTIJ4DJC6MILW2WUXCRQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.110943775 (377h51m8.88158511s) [noise=true , address=104.36.82.9:4007 , piece_id=PWEBGYEIZKF7LMAP5OIPFHI4MUTEESH7AP4YNPR4HZRLQ6W2RK7A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.117461985 (377h51m8.88810332s) [noise=true , address=104.36.82.13:4185 , piece_id=ONP27OBE47FRF65P7WER6TPYUIQV5JL6HFPW27QZWWKIQRW4UZ6Q, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.161579608 (377h51m8.932220933s) [noise=true , address=45.91.116.5:4298 , piece_id=J6246HQ74DVP7OCDYP7RZV6LYMUOHSBZUHPGRRKIFFBMF6H2VJ4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.170088417 (377h51m8.940729752s) [noise=true , address=45.91.116.4:4937 , piece_id=R2KT7NL4HIHVCAWL4VCHNZXJHD6LBLVLQT5VPTZNOIMHSSOKCFCA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.170743521 (377h51m8.941384846s) [noise=true , address=45.91.116.13:4154 , piece_id=ONJJYHN6V4GBSEJC6HQAVW6D4B3X3HMZYINJY3QTLPVH3CXDZXWQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.176248322 (377h51m8.946889647s) [noise=true , address=45.91.116.14:4189 , piece_id=C4T6UUDVQLCBC4ARQ5PYUTC2STW235PJVERKBETDJ33ZDEEZDCWA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.183619354 (377h51m8.954260679s) [noise=true , address=45.91.116.8:4045 , piece_id=5675275I45RIW7LNXNS2K5AH32S7NS5F6M7GYBF32O5O7QOXN57A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.236599983 (377h51m9.007241318s) [noise=true , address=77.90.31.16:4735 , piece_id=AUOIUCZIJLLNGM37ZWT5DYQWLDXJF363XAMQ2ZFEIAHL6AEAERNQ, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.242137387 (377h51m9.012778712s) [noise=true , address=77.90.31.16:4733 , piece_id=LZWLNT7UKJPTJN6CZ5UH6DZ2MYLQE5OTHMQETQKIODJMAPX73IMA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.242995759 (377h51m9.013637094s) [noise=true , address=45.91.116.9:4261 , piece_id=4BSAVF57L56GOJEYLFQBP7CJWVBAP6JMBA7E5MWU5OXZT6R5EXOA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: <nil> Jul 9 03:01:34.243008334 (377h51m9.013649669s) [seg_pos={0 0}, upload=302293]: Segment reached optimal threshold of 13 pieces. Jul 9 03:01:34.243124386 (377h51m9.013765721s) [noise=true , address=45.91.116.8:4047 , piece_id=66XRNA3S672IDAIB6ZIVULIDI23Y24T47OTK6AWASYZJJTZCZUAA, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243179531 (377h51m9.013820866s) [noise=true , address=104.36.82.8:4969 , piece_id=Z5STRISGA5NLFDIDRR2F6DWSYJUQ5X5N7T7MM3ZYMPO4U4SQVR4A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243274663 (377h51m9.013915998s) [noise=true , address=45.91.116.12:4227 , piece_id=2WJA3G36YXWLDNKRZV3HYZCSTTJMZ7PQ4JUK3OWAIBR2IS5OAN5A, seg_pos={0 0}, upload=302293]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Jul 9 03:01:34.243323757 (377h51m9.013965092s) [seg_pos={0 0}, upload=302293]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 9 03:01:34.243412717 (377h51m9.014054052s) [upload=302293]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 03:01:34.243417376 (377h51m9.014058711s) [upload=302293]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 03:01:34.538397355 (377h51m9.30903869s) [upload=302293]: Done uploading segments... Jul 9 03:01:34.538417173 (377h51m9.309058508s) [upload=302293]: Upload finished. err: <nil> Duration: 5.745108935s core uplink us-select-1

1
...
8405
8406
8407
8408
8409
...
8695