Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 430442 reports (Page 8290 of 8609)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.71130655s FAIL (2025-07-09 22:12:29 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 5.711333682s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.434324746s FAIL (2025-07-09 22:12:11 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 5.434354663s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X.X.........] duration: 6.459016599s FAIL (2025-07-09 22:12:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.459042677s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...........X..X.X...X...X.......X............X.X............] duration: 24.04301712s FAIL (2025-07-09 22:11:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 17505]: Done starting upload Jul 9 22:01:35.031454414 (396h51m9.802095779s) [upload=317505]: Uploading segments... Jul 9 22:01:35.031760129 (396h51m9.802401504s) [upload=317505]: Got next segment. Inline: false Jul 9 22:01:35.031826716 (396h51m9.802468081s) [upload=317505]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:01:35.031834370 (396h51m9.802475725s) [upload=317505]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:01:35.482978326 (396h51m10.253619662s) [seg_pos={0 0}, upload=317505]: Begin upload segment... Jul 9 22:01:35.483027780 (396h51m10.253669115s) [seg_pos={0 0}, upload=317505]: Done begin upload segment. Jul 9 22:01:35.483053139 (396h51m10.253694474s) [noise=true , address=77.90.31.16:4741 , piece_id=XGSTR3METENXDLWEX5V65WJJPVLYCTF5FNN6OWOLNQ35EMYF4YBQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483088858 (396h51m10.253730193s) [noise=true , address=104.36.82.17:4340 , piece_id=7SQQWWMV5Z6JVSOQKHJXCJAECAPNYVSXTNCWH3I2JLF6GEXAUWHQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483512137 (396h51m10.254153472s) [noise=true , address=79.127.219.38:4013 , piece_id=IQPTM37ESAZZJELY4RLPBLBMEFPFYXP63SDVK47P7Y3HN3JKIYEQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483526023 (396h51m10.254167359s) [noise=true , address=45.91.95.3:4018 , piece_id=WEMOWEK3L6A3GLWZBG4BN4JTP6PWPYZ5ATAODXEL6AAJHE2XA7AA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483777826 (396h51m10.254419161s) [noise=true , address=104.36.82.38:4709 , piece_id=CB52JNMPPONRIHQXMBCL4NVC23ONSTA2T4ZVYEU67A3VQZXMMLWA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483903696 (396h51m10.254545031s) [noise=true , address=77.90.31.17:4789 , piece_id=XS44AT7VLC7G6VVVV4HVZHNT7YTYWLAC2EOPDPDSUQ6BNMXQL2EA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484036110 (396h51m10.254677435s) [noise=true , address=45.91.95.5:4033 , piece_id=GOBAVQTDDZOWQHJGUVIKCRNPMEC7WC5GW5BYEQVAYDJKQK7KXAIQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484318500 (396h51m10.254959835s) [noise=true , address=77.90.31.10:4547 , piece_id=GX2EF5YML4GXZ2GH5J54YBV7CGE5JKMC63MECWAMDCZYUA6QUE5A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484411849 (396h51m10.255053184s) [noise=true , address=77.90.31.14:4674 , piece_id=6HVO3E5TKEL5JSCA3AKOS3F6NCPD34L3ACAJGCBKDXMJQNBK2Y2Q, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484600008 (396h51m10.255241343s) [noise=true , address=77.90.31.13:4638 , piece_id=OLHXLVAS74AXSME7T242FXYDRNY3TAASGFWRZG2DQ7Y7TY45M6IQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484754364 (396h51m10.255395699s) [noise=true , address=77.90.31.18:4834 , piece_id=7BABB2K6Z5ROMH7GL4ZS3NMMTCMMKGAK3YQRW45BOQ2YSLVZ6JQQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484809600 (396h51m10.255450935s) [noise=true , address=45.91.95.2:4008 , piece_id=WGWH2DZO6DGMDZBAANIDLWHR6MOCHHUZQREZAZZGGS4SE3ILP74A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485011466 (396h51m10.255652791s) [noise=true , address=45.91.95.4:4028 , piece_id=LNOPV7BHCMO3EAEXSYHXMHR4GJ3277WDWMFZMBE6DAP32RV2G26Q, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485075679 (396h51m10.255717014s) [noise=true , address=79.127.220.98:4416 , piece_id=JMUXH2XXYATER2XYXWQMNOWSMVYVSXKCYGH5MHRIZ2XU2T4WKQZQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485387596 (396h51m10.256028931s) [noise=true , address=77.90.31.21:4912 , piece_id=5ILPD7GX2AGAQXJLQQ5MLKVQOB6I5Q37RQV33EGGSUZIS57WIY2A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485694533 (396h51m10.256335868s) [noise=true , address=104.36.82.60:4339 , piece_id=VFM6GGQB3WJ3CWFXPSJN5BL2FQUTLNHURJRFBUT3XAD5CI6H5YYA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.486011620 (396h51m10.256652955s) [noise=true , address=104.36.82.36:4628 , piece_id=WW4Z3U3GIMLRET2K2KJ4D255ROXIQH5DZBZOJQMGU62YEXKZ6G3A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.486306593 (396h51m10.256947928s) [noise=true , address=104.36.82.44:4875 , piece_id=NPZIJRT65NIYURDZOEQVZW5EJPCDGPL4RVKRQJ4EURFQDJXQZJMA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:36.891865578 (396h51m11.662506913s) [upload=317505]: Next returned nil segment Jul 9 22:01:36.891886257 (396h51m11.662527582s) [upload=317505]: Waiting for error group managing segments... Jul 9 22:01:37.305485395 (396h51m12.07612673s) [noise=true , address=79.127.220.98:4416 , piece_id=JMUXH2XXYATER2XYXWQMNOWSMVYVSXKCYGH5MHRIZ2XU2T4WKQZQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.473014492 (396h51m12.243655827s) [noise=true , address=79.127.219.38:4013 , piece_id=IQPTM37ESAZZJELY4RLPBLBMEFPFYXP63SDVK47P7Y3HN3JKIYEQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.476121286 (396h51m12.246762621s) [noise=true , address=77.90.31.10:4547 , piece_id=GX2EF5YML4GXZ2GH5J54YBV7CGE5JKMC63MECWAMDCZYUA6QUE5A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.486869289 (396h51m12.257510624s) [noise=true , address=77.90.31.17:4789 , piece_id=XS44AT7VLC7G6VVVV4HVZHNT7YTYWLAC2EOPDPDSUQ6BNMXQL2EA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.491485340 (396h51m12.262126695s) [noise=true , address=77.90.31.18:4834 , piece_id=7BABB2K6Z5ROMH7GL4ZS3NMMTCMMKGAK3YQRW45BOQ2YSLVZ6JQQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.491802476 (396h51m12.262443811s) [noise=true , address=77.90.31.16:4741 , piece_id=XGSTR3METENXDLWEX5V65WJJPVLYCTF5FNN6OWOLNQ35EMYF4YBQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.492708459 (396h51m12.263349785s) [noise=true , address=77.90.31.21:4912 , piece_id=5ILPD7GX2AGAQXJLQQ5MLKVQOB6I5Q37RQV33EGGSUZIS57WIY2A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.495223731 (396h51m12.265865087s) [noise=true , address=77.90.31.14:4674 , piece_id=6HVO3E5TKEL5JSCA3AKOS3F6NCPD34L3ACAJGCBKDXMJQNBK2Y2Q, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.496454086 (396h51m12.267095441s) [noise=true , address=77.90.31.13:4638 , piece_id=OLHXLVAS74AXSME7T242FXYDRNY3TAASGFWRZG2DQ7Y7TY45M6IQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:02.845874616 (396h51m37.616515951s) [noise=true , address=45.91.95.4:4028 , piece_id=LNOPV7BHCMO3EAEXSYHXMHR4GJ3277WDWMFZMBE6DAP32RV2G26Q, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:08.932651945 (396h51m43.70329328s) [noise=true , address=104.36.82.44:4875 , piece_id=NPZIJRT65NIYURDZOEQVZW5EJPCDGPL4RVKRQJ4EURFQDJXQZJMA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:17.144956582 (396h51m51.915597917s) [noise=true , address=104.36.82.17:4340 , piece_id=7SQQWWMV5Z6JVSOQKHJXCJAECAPNYVSXTNCWH3I2JLF6GEXAUWHQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:33.266596631 (396h52m8.037237966s) [noise=true , address=45.91.95.5:4033 , piece_id=GOBAVQTDDZOWQHJGUVIKCRNPMEC7WC5GW5BYEQVAYDJKQK7KXAIQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:34.385712938 (396h52m9.156354273s) [noise=true , address=104.36.82.36:4628 , piece_id=WW4Z3U3GIMLRET2K2KJ4D255ROXIQH5DZBZOJQMGU62YEXKZ6G3A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385742234 (396h52m9.156383569s) [noise=true , address=104.36.82.38:4709 , piece_id=CB52JNMPPONRIHQXMBCL4NVC23ONSTA2T4ZVYEU67A3VQZXMMLWA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1Ad77dgyCKuYL8o9Ac1WrMfk7pG1mgkakcVXwZa2QHD6KG5sdH, address:104.36.82.38:4709): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385815304 (396h52m9.156456639s) [noise=true , address=45.91.95.3:4018 , piece_id=WEMOWEK3L6A3GLWZBG4BN4JTP6PWPYZ5ATAODXEL6AAJHE2XA7AA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:12qwhpXGKbp4BZ5xTgzYZyw297QcYcrTWo3zUeXVecXsfRqxB6c, address:45.91.95.3:4018): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385853136 (396h52m9.156494471s) [noise=true , address=104.36.82.60:4339 , piece_id=VFM6GGQB3WJ3CWFXPSJN5BL2FQUTLNHURJRFBUT3XAD5CI6H5YYA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385905035 (396h52m9.15654637s) [noise=true , address=45.91.95.2:4008 , piece_id=WGWH2DZO6DGMDZBAANIDLWHR6MOCHHUZQREZAZZGGS4SE3ILP74A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1WTvNJu1X2JaZAT9x58hxgmx849MyJWXWVsbwe8SfQexjdnJAF, address:45.91.95.2:4008): protocol: write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection; write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection Jul 9 22:02:34.385937026 (396h52m9.156578421s) [seg_pos={0 0}, upload=317505]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 22:02:34.637872527 (396h52m9.408513862s) [upload=317505]: Done uploading segments... Jul 9 22:02:34.637897874 (396h52m9.40853921s) [upload=317505]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 24.043051385s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [...............X..X....X........X.........X................X] duration: 1m0.000669749s FAIL (2025-07-09 22:11:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000703743s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.228129673s FAIL (2025-07-09 22:11:23 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 5.228157567s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X.X........] duration: 6.209198316s FAIL (2025-07-09 22:11:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.209218884s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [............X..X.X...X...X.......X............X.X...........] duration: 25.34160287s FAIL (2025-07-09 22:10:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 17505]: Done starting upload Jul 9 22:01:35.031454414 (396h51m9.802095779s) [upload=317505]: Uploading segments... Jul 9 22:01:35.031760129 (396h51m9.802401504s) [upload=317505]: Got next segment. Inline: false Jul 9 22:01:35.031826716 (396h51m9.802468081s) [upload=317505]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:01:35.031834370 (396h51m9.802475725s) [upload=317505]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:01:35.482978326 (396h51m10.253619662s) [seg_pos={0 0}, upload=317505]: Begin upload segment... Jul 9 22:01:35.483027780 (396h51m10.253669115s) [seg_pos={0 0}, upload=317505]: Done begin upload segment. Jul 9 22:01:35.483053139 (396h51m10.253694474s) [noise=true , address=77.90.31.16:4741 , piece_id=XGSTR3METENXDLWEX5V65WJJPVLYCTF5FNN6OWOLNQ35EMYF4YBQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483088858 (396h51m10.253730193s) [noise=true , address=104.36.82.17:4340 , piece_id=7SQQWWMV5Z6JVSOQKHJXCJAECAPNYVSXTNCWH3I2JLF6GEXAUWHQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483512137 (396h51m10.254153472s) [noise=true , address=79.127.219.38:4013 , piece_id=IQPTM37ESAZZJELY4RLPBLBMEFPFYXP63SDVK47P7Y3HN3JKIYEQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483526023 (396h51m10.254167359s) [noise=true , address=45.91.95.3:4018 , piece_id=WEMOWEK3L6A3GLWZBG4BN4JTP6PWPYZ5ATAODXEL6AAJHE2XA7AA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483777826 (396h51m10.254419161s) [noise=true , address=104.36.82.38:4709 , piece_id=CB52JNMPPONRIHQXMBCL4NVC23ONSTA2T4ZVYEU67A3VQZXMMLWA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483903696 (396h51m10.254545031s) [noise=true , address=77.90.31.17:4789 , piece_id=XS44AT7VLC7G6VVVV4HVZHNT7YTYWLAC2EOPDPDSUQ6BNMXQL2EA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484036110 (396h51m10.254677435s) [noise=true , address=45.91.95.5:4033 , piece_id=GOBAVQTDDZOWQHJGUVIKCRNPMEC7WC5GW5BYEQVAYDJKQK7KXAIQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484318500 (396h51m10.254959835s) [noise=true , address=77.90.31.10:4547 , piece_id=GX2EF5YML4GXZ2GH5J54YBV7CGE5JKMC63MECWAMDCZYUA6QUE5A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484411849 (396h51m10.255053184s) [noise=true , address=77.90.31.14:4674 , piece_id=6HVO3E5TKEL5JSCA3AKOS3F6NCPD34L3ACAJGCBKDXMJQNBK2Y2Q, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484600008 (396h51m10.255241343s) [noise=true , address=77.90.31.13:4638 , piece_id=OLHXLVAS74AXSME7T242FXYDRNY3TAASGFWRZG2DQ7Y7TY45M6IQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484754364 (396h51m10.255395699s) [noise=true , address=77.90.31.18:4834 , piece_id=7BABB2K6Z5ROMH7GL4ZS3NMMTCMMKGAK3YQRW45BOQ2YSLVZ6JQQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484809600 (396h51m10.255450935s) [noise=true , address=45.91.95.2:4008 , piece_id=WGWH2DZO6DGMDZBAANIDLWHR6MOCHHUZQREZAZZGGS4SE3ILP74A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485011466 (396h51m10.255652791s) [noise=true , address=45.91.95.4:4028 , piece_id=LNOPV7BHCMO3EAEXSYHXMHR4GJ3277WDWMFZMBE6DAP32RV2G26Q, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485075679 (396h51m10.255717014s) [noise=true , address=79.127.220.98:4416 , piece_id=JMUXH2XXYATER2XYXWQMNOWSMVYVSXKCYGH5MHRIZ2XU2T4WKQZQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485387596 (396h51m10.256028931s) [noise=true , address=77.90.31.21:4912 , piece_id=5ILPD7GX2AGAQXJLQQ5MLKVQOB6I5Q37RQV33EGGSUZIS57WIY2A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485694533 (396h51m10.256335868s) [noise=true , address=104.36.82.60:4339 , piece_id=VFM6GGQB3WJ3CWFXPSJN5BL2FQUTLNHURJRFBUT3XAD5CI6H5YYA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.486011620 (396h51m10.256652955s) [noise=true , address=104.36.82.36:4628 , piece_id=WW4Z3U3GIMLRET2K2KJ4D255ROXIQH5DZBZOJQMGU62YEXKZ6G3A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.486306593 (396h51m10.256947928s) [noise=true , address=104.36.82.44:4875 , piece_id=NPZIJRT65NIYURDZOEQVZW5EJPCDGPL4RVKRQJ4EURFQDJXQZJMA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:36.891865578 (396h51m11.662506913s) [upload=317505]: Next returned nil segment Jul 9 22:01:36.891886257 (396h51m11.662527582s) [upload=317505]: Waiting for error group managing segments... Jul 9 22:01:37.305485395 (396h51m12.07612673s) [noise=true , address=79.127.220.98:4416 , piece_id=JMUXH2XXYATER2XYXWQMNOWSMVYVSXKCYGH5MHRIZ2XU2T4WKQZQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.473014492 (396h51m12.243655827s) [noise=true , address=79.127.219.38:4013 , piece_id=IQPTM37ESAZZJELY4RLPBLBMEFPFYXP63SDVK47P7Y3HN3JKIYEQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.476121286 (396h51m12.246762621s) [noise=true , address=77.90.31.10:4547 , piece_id=GX2EF5YML4GXZ2GH5J54YBV7CGE5JKMC63MECWAMDCZYUA6QUE5A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.486869289 (396h51m12.257510624s) [noise=true , address=77.90.31.17:4789 , piece_id=XS44AT7VLC7G6VVVV4HVZHNT7YTYWLAC2EOPDPDSUQ6BNMXQL2EA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.491485340 (396h51m12.262126695s) [noise=true , address=77.90.31.18:4834 , piece_id=7BABB2K6Z5ROMH7GL4ZS3NMMTCMMKGAK3YQRW45BOQ2YSLVZ6JQQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.491802476 (396h51m12.262443811s) [noise=true , address=77.90.31.16:4741 , piece_id=XGSTR3METENXDLWEX5V65WJJPVLYCTF5FNN6OWOLNQ35EMYF4YBQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.492708459 (396h51m12.263349785s) [noise=true , address=77.90.31.21:4912 , piece_id=5ILPD7GX2AGAQXJLQQ5MLKVQOB6I5Q37RQV33EGGSUZIS57WIY2A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.495223731 (396h51m12.265865087s) [noise=true , address=77.90.31.14:4674 , piece_id=6HVO3E5TKEL5JSCA3AKOS3F6NCPD34L3ACAJGCBKDXMJQNBK2Y2Q, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.496454086 (396h51m12.267095441s) [noise=true , address=77.90.31.13:4638 , piece_id=OLHXLVAS74AXSME7T242FXYDRNY3TAASGFWRZG2DQ7Y7TY45M6IQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:02.845874616 (396h51m37.616515951s) [noise=true , address=45.91.95.4:4028 , piece_id=LNOPV7BHCMO3EAEXSYHXMHR4GJ3277WDWMFZMBE6DAP32RV2G26Q, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:08.932651945 (396h51m43.70329328s) [noise=true , address=104.36.82.44:4875 , piece_id=NPZIJRT65NIYURDZOEQVZW5EJPCDGPL4RVKRQJ4EURFQDJXQZJMA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:17.144956582 (396h51m51.915597917s) [noise=true , address=104.36.82.17:4340 , piece_id=7SQQWWMV5Z6JVSOQKHJXCJAECAPNYVSXTNCWH3I2JLF6GEXAUWHQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:33.266596631 (396h52m8.037237966s) [noise=true , address=45.91.95.5:4033 , piece_id=GOBAVQTDDZOWQHJGUVIKCRNPMEC7WC5GW5BYEQVAYDJKQK7KXAIQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:34.385712938 (396h52m9.156354273s) [noise=true , address=104.36.82.36:4628 , piece_id=WW4Z3U3GIMLRET2K2KJ4D255ROXIQH5DZBZOJQMGU62YEXKZ6G3A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385742234 (396h52m9.156383569s) [noise=true , address=104.36.82.38:4709 , piece_id=CB52JNMPPONRIHQXMBCL4NVC23ONSTA2T4ZVYEU67A3VQZXMMLWA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1Ad77dgyCKuYL8o9Ac1WrMfk7pG1mgkakcVXwZa2QHD6KG5sdH, address:104.36.82.38:4709): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385815304 (396h52m9.156456639s) [noise=true , address=45.91.95.3:4018 , piece_id=WEMOWEK3L6A3GLWZBG4BN4JTP6PWPYZ5ATAODXEL6AAJHE2XA7AA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:12qwhpXGKbp4BZ5xTgzYZyw297QcYcrTWo3zUeXVecXsfRqxB6c, address:45.91.95.3:4018): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385853136 (396h52m9.156494471s) [noise=true , address=104.36.82.60:4339 , piece_id=VFM6GGQB3WJ3CWFXPSJN5BL2FQUTLNHURJRFBUT3XAD5CI6H5YYA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385905035 (396h52m9.15654637s) [noise=true , address=45.91.95.2:4008 , piece_id=WGWH2DZO6DGMDZBAANIDLWHR6MOCHHUZQREZAZZGGS4SE3ILP74A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1WTvNJu1X2JaZAT9x58hxgmx849MyJWXWVsbwe8SfQexjdnJAF, address:45.91.95.2:4008): protocol: write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection; write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection Jul 9 22:02:34.385937026 (396h52m9.156578421s) [seg_pos={0 0}, upload=317505]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 22:02:34.637872527 (396h52m9.408513862s) [upload=317505]: Done uploading segments... Jul 9 22:02:34.637897874 (396h52m9.40853921s) [upload=317505]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 25.341625904s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [X...............X..X....X........X.........X................] duration: 55.675156746s FAIL (2025-07-09 22:10:54 UTC) orbiter warning: context deadline exceeded Duration: 55.675209136s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.51708725s FAIL (2025-07-09 22:10:32 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 5.517110947s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X.X.......] duration: 6.190524465s FAIL (2025-07-09 22:10:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.190547218s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.............X..X.X...X...X.......X............X.X..........] duration: 40.7907348s FAIL (2025-07-09 22:10:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 17505]: Done starting upload Jul 9 22:01:35.031454414 (396h51m9.802095779s) [upload=317505]: Uploading segments... Jul 9 22:01:35.031760129 (396h51m9.802401504s) [upload=317505]: Got next segment. Inline: false Jul 9 22:01:35.031826716 (396h51m9.802468081s) [upload=317505]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:01:35.031834370 (396h51m9.802475725s) [upload=317505]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:01:35.482978326 (396h51m10.253619662s) [seg_pos={0 0}, upload=317505]: Begin upload segment... Jul 9 22:01:35.483027780 (396h51m10.253669115s) [seg_pos={0 0}, upload=317505]: Done begin upload segment. Jul 9 22:01:35.483053139 (396h51m10.253694474s) [noise=true , address=77.90.31.16:4741 , piece_id=XGSTR3METENXDLWEX5V65WJJPVLYCTF5FNN6OWOLNQ35EMYF4YBQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483088858 (396h51m10.253730193s) [noise=true , address=104.36.82.17:4340 , piece_id=7SQQWWMV5Z6JVSOQKHJXCJAECAPNYVSXTNCWH3I2JLF6GEXAUWHQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483512137 (396h51m10.254153472s) [noise=true , address=79.127.219.38:4013 , piece_id=IQPTM37ESAZZJELY4RLPBLBMEFPFYXP63SDVK47P7Y3HN3JKIYEQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483526023 (396h51m10.254167359s) [noise=true , address=45.91.95.3:4018 , piece_id=WEMOWEK3L6A3GLWZBG4BN4JTP6PWPYZ5ATAODXEL6AAJHE2XA7AA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483777826 (396h51m10.254419161s) [noise=true , address=104.36.82.38:4709 , piece_id=CB52JNMPPONRIHQXMBCL4NVC23ONSTA2T4ZVYEU67A3VQZXMMLWA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483903696 (396h51m10.254545031s) [noise=true , address=77.90.31.17:4789 , piece_id=XS44AT7VLC7G6VVVV4HVZHNT7YTYWLAC2EOPDPDSUQ6BNMXQL2EA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484036110 (396h51m10.254677435s) [noise=true , address=45.91.95.5:4033 , piece_id=GOBAVQTDDZOWQHJGUVIKCRNPMEC7WC5GW5BYEQVAYDJKQK7KXAIQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484318500 (396h51m10.254959835s) [noise=true , address=77.90.31.10:4547 , piece_id=GX2EF5YML4GXZ2GH5J54YBV7CGE5JKMC63MECWAMDCZYUA6QUE5A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484411849 (396h51m10.255053184s) [noise=true , address=77.90.31.14:4674 , piece_id=6HVO3E5TKEL5JSCA3AKOS3F6NCPD34L3ACAJGCBKDXMJQNBK2Y2Q, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484600008 (396h51m10.255241343s) [noise=true , address=77.90.31.13:4638 , piece_id=OLHXLVAS74AXSME7T242FXYDRNY3TAASGFWRZG2DQ7Y7TY45M6IQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484754364 (396h51m10.255395699s) [noise=true , address=77.90.31.18:4834 , piece_id=7BABB2K6Z5ROMH7GL4ZS3NMMTCMMKGAK3YQRW45BOQ2YSLVZ6JQQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484809600 (396h51m10.255450935s) [noise=true , address=45.91.95.2:4008 , piece_id=WGWH2DZO6DGMDZBAANIDLWHR6MOCHHUZQREZAZZGGS4SE3ILP74A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485011466 (396h51m10.255652791s) [noise=true , address=45.91.95.4:4028 , piece_id=LNOPV7BHCMO3EAEXSYHXMHR4GJ3277WDWMFZMBE6DAP32RV2G26Q, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485075679 (396h51m10.255717014s) [noise=true , address=79.127.220.98:4416 , piece_id=JMUXH2XXYATER2XYXWQMNOWSMVYVSXKCYGH5MHRIZ2XU2T4WKQZQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485387596 (396h51m10.256028931s) [noise=true , address=77.90.31.21:4912 , piece_id=5ILPD7GX2AGAQXJLQQ5MLKVQOB6I5Q37RQV33EGGSUZIS57WIY2A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485694533 (396h51m10.256335868s) [noise=true , address=104.36.82.60:4339 , piece_id=VFM6GGQB3WJ3CWFXPSJN5BL2FQUTLNHURJRFBUT3XAD5CI6H5YYA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.486011620 (396h51m10.256652955s) [noise=true , address=104.36.82.36:4628 , piece_id=WW4Z3U3GIMLRET2K2KJ4D255ROXIQH5DZBZOJQMGU62YEXKZ6G3A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.486306593 (396h51m10.256947928s) [noise=true , address=104.36.82.44:4875 , piece_id=NPZIJRT65NIYURDZOEQVZW5EJPCDGPL4RVKRQJ4EURFQDJXQZJMA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:36.891865578 (396h51m11.662506913s) [upload=317505]: Next returned nil segment Jul 9 22:01:36.891886257 (396h51m11.662527582s) [upload=317505]: Waiting for error group managing segments... Jul 9 22:01:37.305485395 (396h51m12.07612673s) [noise=true , address=79.127.220.98:4416 , piece_id=JMUXH2XXYATER2XYXWQMNOWSMVYVSXKCYGH5MHRIZ2XU2T4WKQZQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.473014492 (396h51m12.243655827s) [noise=true , address=79.127.219.38:4013 , piece_id=IQPTM37ESAZZJELY4RLPBLBMEFPFYXP63SDVK47P7Y3HN3JKIYEQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.476121286 (396h51m12.246762621s) [noise=true , address=77.90.31.10:4547 , piece_id=GX2EF5YML4GXZ2GH5J54YBV7CGE5JKMC63MECWAMDCZYUA6QUE5A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.486869289 (396h51m12.257510624s) [noise=true , address=77.90.31.17:4789 , piece_id=XS44AT7VLC7G6VVVV4HVZHNT7YTYWLAC2EOPDPDSUQ6BNMXQL2EA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.491485340 (396h51m12.262126695s) [noise=true , address=77.90.31.18:4834 , piece_id=7BABB2K6Z5ROMH7GL4ZS3NMMTCMMKGAK3YQRW45BOQ2YSLVZ6JQQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.491802476 (396h51m12.262443811s) [noise=true , address=77.90.31.16:4741 , piece_id=XGSTR3METENXDLWEX5V65WJJPVLYCTF5FNN6OWOLNQ35EMYF4YBQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.492708459 (396h51m12.263349785s) [noise=true , address=77.90.31.21:4912 , piece_id=5ILPD7GX2AGAQXJLQQ5MLKVQOB6I5Q37RQV33EGGSUZIS57WIY2A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.495223731 (396h51m12.265865087s) [noise=true , address=77.90.31.14:4674 , piece_id=6HVO3E5TKEL5JSCA3AKOS3F6NCPD34L3ACAJGCBKDXMJQNBK2Y2Q, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.496454086 (396h51m12.267095441s) [noise=true , address=77.90.31.13:4638 , piece_id=OLHXLVAS74AXSME7T242FXYDRNY3TAASGFWRZG2DQ7Y7TY45M6IQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:02.845874616 (396h51m37.616515951s) [noise=true , address=45.91.95.4:4028 , piece_id=LNOPV7BHCMO3EAEXSYHXMHR4GJ3277WDWMFZMBE6DAP32RV2G26Q, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:08.932651945 (396h51m43.70329328s) [noise=true , address=104.36.82.44:4875 , piece_id=NPZIJRT65NIYURDZOEQVZW5EJPCDGPL4RVKRQJ4EURFQDJXQZJMA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:17.144956582 (396h51m51.915597917s) [noise=true , address=104.36.82.17:4340 , piece_id=7SQQWWMV5Z6JVSOQKHJXCJAECAPNYVSXTNCWH3I2JLF6GEXAUWHQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:33.266596631 (396h52m8.037237966s) [noise=true , address=45.91.95.5:4033 , piece_id=GOBAVQTDDZOWQHJGUVIKCRNPMEC7WC5GW5BYEQVAYDJKQK7KXAIQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:34.385712938 (396h52m9.156354273s) [noise=true , address=104.36.82.36:4628 , piece_id=WW4Z3U3GIMLRET2K2KJ4D255ROXIQH5DZBZOJQMGU62YEXKZ6G3A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385742234 (396h52m9.156383569s) [noise=true , address=104.36.82.38:4709 , piece_id=CB52JNMPPONRIHQXMBCL4NVC23ONSTA2T4ZVYEU67A3VQZXMMLWA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1Ad77dgyCKuYL8o9Ac1WrMfk7pG1mgkakcVXwZa2QHD6KG5sdH, address:104.36.82.38:4709): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385815304 (396h52m9.156456639s) [noise=true , address=45.91.95.3:4018 , piece_id=WEMOWEK3L6A3GLWZBG4BN4JTP6PWPYZ5ATAODXEL6AAJHE2XA7AA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:12qwhpXGKbp4BZ5xTgzYZyw297QcYcrTWo3zUeXVecXsfRqxB6c, address:45.91.95.3:4018): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385853136 (396h52m9.156494471s) [noise=true , address=104.36.82.60:4339 , piece_id=VFM6GGQB3WJ3CWFXPSJN5BL2FQUTLNHURJRFBUT3XAD5CI6H5YYA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385905035 (396h52m9.15654637s) [noise=true , address=45.91.95.2:4008 , piece_id=WGWH2DZO6DGMDZBAANIDLWHR6MOCHHUZQREZAZZGGS4SE3ILP74A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1WTvNJu1X2JaZAT9x58hxgmx849MyJWXWVsbwe8SfQexjdnJAF, address:45.91.95.2:4008): protocol: write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection; write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection Jul 9 22:02:34.385937026 (396h52m9.156578421s) [seg_pos={0 0}, upload=317505]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 22:02:34.637872527 (396h52m9.408513862s) [upload=317505]: Done uploading segments... Jul 9 22:02:34.637897874 (396h52m9.40853921s) [upload=317505]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 40.790758757s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.103264376s FAIL (2025-07-09 22:09:37 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 5.103291328s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X.X......] duration: 5.899155266s FAIL (2025-07-09 22:09:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.899196155s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.X...............X..X....X........X.........X...............] duration: 39.62929445s FAIL (2025-07-09 22:09:27 UTC) orbiter warning: context deadline exceeded Duration: 39.629311473s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [..............X..X.X...X...X.......X............X.X.........] duration: 27.277588171s FAIL (2025-07-09 22:09:15 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 17505]: Done starting upload Jul 9 22:01:35.031454414 (396h51m9.802095779s) [upload=317505]: Uploading segments... Jul 9 22:01:35.031760129 (396h51m9.802401504s) [upload=317505]: Got next segment. Inline: false Jul 9 22:01:35.031826716 (396h51m9.802468081s) [upload=317505]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:01:35.031834370 (396h51m9.802475725s) [upload=317505]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:01:35.482978326 (396h51m10.253619662s) [seg_pos={0 0}, upload=317505]: Begin upload segment... Jul 9 22:01:35.483027780 (396h51m10.253669115s) [seg_pos={0 0}, upload=317505]: Done begin upload segment. Jul 9 22:01:35.483053139 (396h51m10.253694474s) [noise=true , address=77.90.31.16:4741 , piece_id=XGSTR3METENXDLWEX5V65WJJPVLYCTF5FNN6OWOLNQ35EMYF4YBQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483088858 (396h51m10.253730193s) [noise=true , address=104.36.82.17:4340 , piece_id=7SQQWWMV5Z6JVSOQKHJXCJAECAPNYVSXTNCWH3I2JLF6GEXAUWHQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483512137 (396h51m10.254153472s) [noise=true , address=79.127.219.38:4013 , piece_id=IQPTM37ESAZZJELY4RLPBLBMEFPFYXP63SDVK47P7Y3HN3JKIYEQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483526023 (396h51m10.254167359s) [noise=true , address=45.91.95.3:4018 , piece_id=WEMOWEK3L6A3GLWZBG4BN4JTP6PWPYZ5ATAODXEL6AAJHE2XA7AA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483777826 (396h51m10.254419161s) [noise=true , address=104.36.82.38:4709 , piece_id=CB52JNMPPONRIHQXMBCL4NVC23ONSTA2T4ZVYEU67A3VQZXMMLWA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483903696 (396h51m10.254545031s) [noise=true , address=77.90.31.17:4789 , piece_id=XS44AT7VLC7G6VVVV4HVZHNT7YTYWLAC2EOPDPDSUQ6BNMXQL2EA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484036110 (396h51m10.254677435s) [noise=true , address=45.91.95.5:4033 , piece_id=GOBAVQTDDZOWQHJGUVIKCRNPMEC7WC5GW5BYEQVAYDJKQK7KXAIQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484318500 (396h51m10.254959835s) [noise=true , address=77.90.31.10:4547 , piece_id=GX2EF5YML4GXZ2GH5J54YBV7CGE5JKMC63MECWAMDCZYUA6QUE5A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484411849 (396h51m10.255053184s) [noise=true , address=77.90.31.14:4674 , piece_id=6HVO3E5TKEL5JSCA3AKOS3F6NCPD34L3ACAJGCBKDXMJQNBK2Y2Q, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484600008 (396h51m10.255241343s) [noise=true , address=77.90.31.13:4638 , piece_id=OLHXLVAS74AXSME7T242FXYDRNY3TAASGFWRZG2DQ7Y7TY45M6IQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484754364 (396h51m10.255395699s) [noise=true , address=77.90.31.18:4834 , piece_id=7BABB2K6Z5ROMH7GL4ZS3NMMTCMMKGAK3YQRW45BOQ2YSLVZ6JQQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484809600 (396h51m10.255450935s) [noise=true , address=45.91.95.2:4008 , piece_id=WGWH2DZO6DGMDZBAANIDLWHR6MOCHHUZQREZAZZGGS4SE3ILP74A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485011466 (396h51m10.255652791s) [noise=true , address=45.91.95.4:4028 , piece_id=LNOPV7BHCMO3EAEXSYHXMHR4GJ3277WDWMFZMBE6DAP32RV2G26Q, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485075679 (396h51m10.255717014s) [noise=true , address=79.127.220.98:4416 , piece_id=JMUXH2XXYATER2XYXWQMNOWSMVYVSXKCYGH5MHRIZ2XU2T4WKQZQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485387596 (396h51m10.256028931s) [noise=true , address=77.90.31.21:4912 , piece_id=5ILPD7GX2AGAQXJLQQ5MLKVQOB6I5Q37RQV33EGGSUZIS57WIY2A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485694533 (396h51m10.256335868s) [noise=true , address=104.36.82.60:4339 , piece_id=VFM6GGQB3WJ3CWFXPSJN5BL2FQUTLNHURJRFBUT3XAD5CI6H5YYA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.486011620 (396h51m10.256652955s) [noise=true , address=104.36.82.36:4628 , piece_id=WW4Z3U3GIMLRET2K2KJ4D255ROXIQH5DZBZOJQMGU62YEXKZ6G3A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.486306593 (396h51m10.256947928s) [noise=true , address=104.36.82.44:4875 , piece_id=NPZIJRT65NIYURDZOEQVZW5EJPCDGPL4RVKRQJ4EURFQDJXQZJMA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:36.891865578 (396h51m11.662506913s) [upload=317505]: Next returned nil segment Jul 9 22:01:36.891886257 (396h51m11.662527582s) [upload=317505]: Waiting for error group managing segments... Jul 9 22:01:37.305485395 (396h51m12.07612673s) [noise=true , address=79.127.220.98:4416 , piece_id=JMUXH2XXYATER2XYXWQMNOWSMVYVSXKCYGH5MHRIZ2XU2T4WKQZQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.473014492 (396h51m12.243655827s) [noise=true , address=79.127.219.38:4013 , piece_id=IQPTM37ESAZZJELY4RLPBLBMEFPFYXP63SDVK47P7Y3HN3JKIYEQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.476121286 (396h51m12.246762621s) [noise=true , address=77.90.31.10:4547 , piece_id=GX2EF5YML4GXZ2GH5J54YBV7CGE5JKMC63MECWAMDCZYUA6QUE5A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.486869289 (396h51m12.257510624s) [noise=true , address=77.90.31.17:4789 , piece_id=XS44AT7VLC7G6VVVV4HVZHNT7YTYWLAC2EOPDPDSUQ6BNMXQL2EA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.491485340 (396h51m12.262126695s) [noise=true , address=77.90.31.18:4834 , piece_id=7BABB2K6Z5ROMH7GL4ZS3NMMTCMMKGAK3YQRW45BOQ2YSLVZ6JQQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.491802476 (396h51m12.262443811s) [noise=true , address=77.90.31.16:4741 , piece_id=XGSTR3METENXDLWEX5V65WJJPVLYCTF5FNN6OWOLNQ35EMYF4YBQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.492708459 (396h51m12.263349785s) [noise=true , address=77.90.31.21:4912 , piece_id=5ILPD7GX2AGAQXJLQQ5MLKVQOB6I5Q37RQV33EGGSUZIS57WIY2A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.495223731 (396h51m12.265865087s) [noise=true , address=77.90.31.14:4674 , piece_id=6HVO3E5TKEL5JSCA3AKOS3F6NCPD34L3ACAJGCBKDXMJQNBK2Y2Q, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.496454086 (396h51m12.267095441s) [noise=true , address=77.90.31.13:4638 , piece_id=OLHXLVAS74AXSME7T242FXYDRNY3TAASGFWRZG2DQ7Y7TY45M6IQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:02.845874616 (396h51m37.616515951s) [noise=true , address=45.91.95.4:4028 , piece_id=LNOPV7BHCMO3EAEXSYHXMHR4GJ3277WDWMFZMBE6DAP32RV2G26Q, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:08.932651945 (396h51m43.70329328s) [noise=true , address=104.36.82.44:4875 , piece_id=NPZIJRT65NIYURDZOEQVZW5EJPCDGPL4RVKRQJ4EURFQDJXQZJMA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:17.144956582 (396h51m51.915597917s) [noise=true , address=104.36.82.17:4340 , piece_id=7SQQWWMV5Z6JVSOQKHJXCJAECAPNYVSXTNCWH3I2JLF6GEXAUWHQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:33.266596631 (396h52m8.037237966s) [noise=true , address=45.91.95.5:4033 , piece_id=GOBAVQTDDZOWQHJGUVIKCRNPMEC7WC5GW5BYEQVAYDJKQK7KXAIQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:34.385712938 (396h52m9.156354273s) [noise=true , address=104.36.82.36:4628 , piece_id=WW4Z3U3GIMLRET2K2KJ4D255ROXIQH5DZBZOJQMGU62YEXKZ6G3A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385742234 (396h52m9.156383569s) [noise=true , address=104.36.82.38:4709 , piece_id=CB52JNMPPONRIHQXMBCL4NVC23ONSTA2T4ZVYEU67A3VQZXMMLWA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1Ad77dgyCKuYL8o9Ac1WrMfk7pG1mgkakcVXwZa2QHD6KG5sdH, address:104.36.82.38:4709): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385815304 (396h52m9.156456639s) [noise=true , address=45.91.95.3:4018 , piece_id=WEMOWEK3L6A3GLWZBG4BN4JTP6PWPYZ5ATAODXEL6AAJHE2XA7AA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:12qwhpXGKbp4BZ5xTgzYZyw297QcYcrTWo3zUeXVecXsfRqxB6c, address:45.91.95.3:4018): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385853136 (396h52m9.156494471s) [noise=true , address=104.36.82.60:4339 , piece_id=VFM6GGQB3WJ3CWFXPSJN5BL2FQUTLNHURJRFBUT3XAD5CI6H5YYA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385905035 (396h52m9.15654637s) [noise=true , address=45.91.95.2:4008 , piece_id=WGWH2DZO6DGMDZBAANIDLWHR6MOCHHUZQREZAZZGGS4SE3ILP74A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1WTvNJu1X2JaZAT9x58hxgmx849MyJWXWVsbwe8SfQexjdnJAF, address:45.91.95.2:4008): protocol: write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection; write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection Jul 9 22:02:34.385937026 (396h52m9.156578421s) [seg_pos={0 0}, upload=317505]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 22:02:34.637872527 (396h52m9.408513862s) [upload=317505]: Done uploading segments... Jul 9 22:02:34.637897874 (396h52m9.40853921s) [upload=317505]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 27.277613579s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.175368143s FAIL (2025-07-09 22:09:13 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 5.175398211s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X.X.....] duration: 11.010604414s FAIL (2025-07-09 22:08:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.010621577s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [...............X..X.X...X...X.......X............X.X........] duration: 38.928414555s FAIL (2025-07-09 22:08:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 17505]: Done starting upload Jul 9 22:01:35.031454414 (396h51m9.802095779s) [upload=317505]: Uploading segments... Jul 9 22:01:35.031760129 (396h51m9.802401504s) [upload=317505]: Got next segment. Inline: false Jul 9 22:01:35.031826716 (396h51m9.802468081s) [upload=317505]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:01:35.031834370 (396h51m9.802475725s) [upload=317505]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:01:35.482978326 (396h51m10.253619662s) [seg_pos={0 0}, upload=317505]: Begin upload segment... Jul 9 22:01:35.483027780 (396h51m10.253669115s) [seg_pos={0 0}, upload=317505]: Done begin upload segment. Jul 9 22:01:35.483053139 (396h51m10.253694474s) [noise=true , address=77.90.31.16:4741 , piece_id=XGSTR3METENXDLWEX5V65WJJPVLYCTF5FNN6OWOLNQ35EMYF4YBQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483088858 (396h51m10.253730193s) [noise=true , address=104.36.82.17:4340 , piece_id=7SQQWWMV5Z6JVSOQKHJXCJAECAPNYVSXTNCWH3I2JLF6GEXAUWHQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483512137 (396h51m10.254153472s) [noise=true , address=79.127.219.38:4013 , piece_id=IQPTM37ESAZZJELY4RLPBLBMEFPFYXP63SDVK47P7Y3HN3JKIYEQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483526023 (396h51m10.254167359s) [noise=true , address=45.91.95.3:4018 , piece_id=WEMOWEK3L6A3GLWZBG4BN4JTP6PWPYZ5ATAODXEL6AAJHE2XA7AA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483777826 (396h51m10.254419161s) [noise=true , address=104.36.82.38:4709 , piece_id=CB52JNMPPONRIHQXMBCL4NVC23ONSTA2T4ZVYEU67A3VQZXMMLWA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483903696 (396h51m10.254545031s) [noise=true , address=77.90.31.17:4789 , piece_id=XS44AT7VLC7G6VVVV4HVZHNT7YTYWLAC2EOPDPDSUQ6BNMXQL2EA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484036110 (396h51m10.254677435s) [noise=true , address=45.91.95.5:4033 , piece_id=GOBAVQTDDZOWQHJGUVIKCRNPMEC7WC5GW5BYEQVAYDJKQK7KXAIQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484318500 (396h51m10.254959835s) [noise=true , address=77.90.31.10:4547 , piece_id=GX2EF5YML4GXZ2GH5J54YBV7CGE5JKMC63MECWAMDCZYUA6QUE5A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484411849 (396h51m10.255053184s) [noise=true , address=77.90.31.14:4674 , piece_id=6HVO3E5TKEL5JSCA3AKOS3F6NCPD34L3ACAJGCBKDXMJQNBK2Y2Q, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484600008 (396h51m10.255241343s) [noise=true , address=77.90.31.13:4638 , piece_id=OLHXLVAS74AXSME7T242FXYDRNY3TAASGFWRZG2DQ7Y7TY45M6IQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484754364 (396h51m10.255395699s) [noise=true , address=77.90.31.18:4834 , piece_id=7BABB2K6Z5ROMH7GL4ZS3NMMTCMMKGAK3YQRW45BOQ2YSLVZ6JQQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484809600 (396h51m10.255450935s) [noise=true , address=45.91.95.2:4008 , piece_id=WGWH2DZO6DGMDZBAANIDLWHR6MOCHHUZQREZAZZGGS4SE3ILP74A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485011466 (396h51m10.255652791s) [noise=true , address=45.91.95.4:4028 , piece_id=LNOPV7BHCMO3EAEXSYHXMHR4GJ3277WDWMFZMBE6DAP32RV2G26Q, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485075679 (396h51m10.255717014s) [noise=true , address=79.127.220.98:4416 , piece_id=JMUXH2XXYATER2XYXWQMNOWSMVYVSXKCYGH5MHRIZ2XU2T4WKQZQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485387596 (396h51m10.256028931s) [noise=true , address=77.90.31.21:4912 , piece_id=5ILPD7GX2AGAQXJLQQ5MLKVQOB6I5Q37RQV33EGGSUZIS57WIY2A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485694533 (396h51m10.256335868s) [noise=true , address=104.36.82.60:4339 , piece_id=VFM6GGQB3WJ3CWFXPSJN5BL2FQUTLNHURJRFBUT3XAD5CI6H5YYA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.486011620 (396h51m10.256652955s) [noise=true , address=104.36.82.36:4628 , piece_id=WW4Z3U3GIMLRET2K2KJ4D255ROXIQH5DZBZOJQMGU62YEXKZ6G3A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.486306593 (396h51m10.256947928s) [noise=true , address=104.36.82.44:4875 , piece_id=NPZIJRT65NIYURDZOEQVZW5EJPCDGPL4RVKRQJ4EURFQDJXQZJMA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:36.891865578 (396h51m11.662506913s) [upload=317505]: Next returned nil segment Jul 9 22:01:36.891886257 (396h51m11.662527582s) [upload=317505]: Waiting for error group managing segments... Jul 9 22:01:37.305485395 (396h51m12.07612673s) [noise=true , address=79.127.220.98:4416 , piece_id=JMUXH2XXYATER2XYXWQMNOWSMVYVSXKCYGH5MHRIZ2XU2T4WKQZQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.473014492 (396h51m12.243655827s) [noise=true , address=79.127.219.38:4013 , piece_id=IQPTM37ESAZZJELY4RLPBLBMEFPFYXP63SDVK47P7Y3HN3JKIYEQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.476121286 (396h51m12.246762621s) [noise=true , address=77.90.31.10:4547 , piece_id=GX2EF5YML4GXZ2GH5J54YBV7CGE5JKMC63MECWAMDCZYUA6QUE5A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.486869289 (396h51m12.257510624s) [noise=true , address=77.90.31.17:4789 , piece_id=XS44AT7VLC7G6VVVV4HVZHNT7YTYWLAC2EOPDPDSUQ6BNMXQL2EA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.491485340 (396h51m12.262126695s) [noise=true , address=77.90.31.18:4834 , piece_id=7BABB2K6Z5ROMH7GL4ZS3NMMTCMMKGAK3YQRW45BOQ2YSLVZ6JQQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.491802476 (396h51m12.262443811s) [noise=true , address=77.90.31.16:4741 , piece_id=XGSTR3METENXDLWEX5V65WJJPVLYCTF5FNN6OWOLNQ35EMYF4YBQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.492708459 (396h51m12.263349785s) [noise=true , address=77.90.31.21:4912 , piece_id=5ILPD7GX2AGAQXJLQQ5MLKVQOB6I5Q37RQV33EGGSUZIS57WIY2A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.495223731 (396h51m12.265865087s) [noise=true , address=77.90.31.14:4674 , piece_id=6HVO3E5TKEL5JSCA3AKOS3F6NCPD34L3ACAJGCBKDXMJQNBK2Y2Q, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.496454086 (396h51m12.267095441s) [noise=true , address=77.90.31.13:4638 , piece_id=OLHXLVAS74AXSME7T242FXYDRNY3TAASGFWRZG2DQ7Y7TY45M6IQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:02.845874616 (396h51m37.616515951s) [noise=true , address=45.91.95.4:4028 , piece_id=LNOPV7BHCMO3EAEXSYHXMHR4GJ3277WDWMFZMBE6DAP32RV2G26Q, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:08.932651945 (396h51m43.70329328s) [noise=true , address=104.36.82.44:4875 , piece_id=NPZIJRT65NIYURDZOEQVZW5EJPCDGPL4RVKRQJ4EURFQDJXQZJMA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:17.144956582 (396h51m51.915597917s) [noise=true , address=104.36.82.17:4340 , piece_id=7SQQWWMV5Z6JVSOQKHJXCJAECAPNYVSXTNCWH3I2JLF6GEXAUWHQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:33.266596631 (396h52m8.037237966s) [noise=true , address=45.91.95.5:4033 , piece_id=GOBAVQTDDZOWQHJGUVIKCRNPMEC7WC5GW5BYEQVAYDJKQK7KXAIQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:34.385712938 (396h52m9.156354273s) [noise=true , address=104.36.82.36:4628 , piece_id=WW4Z3U3GIMLRET2K2KJ4D255ROXIQH5DZBZOJQMGU62YEXKZ6G3A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385742234 (396h52m9.156383569s) [noise=true , address=104.36.82.38:4709 , piece_id=CB52JNMPPONRIHQXMBCL4NVC23ONSTA2T4ZVYEU67A3VQZXMMLWA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1Ad77dgyCKuYL8o9Ac1WrMfk7pG1mgkakcVXwZa2QHD6KG5sdH, address:104.36.82.38:4709): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385815304 (396h52m9.156456639s) [noise=true , address=45.91.95.3:4018 , piece_id=WEMOWEK3L6A3GLWZBG4BN4JTP6PWPYZ5ATAODXEL6AAJHE2XA7AA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:12qwhpXGKbp4BZ5xTgzYZyw297QcYcrTWo3zUeXVecXsfRqxB6c, address:45.91.95.3:4018): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385853136 (396h52m9.156494471s) [noise=true , address=104.36.82.60:4339 , piece_id=VFM6GGQB3WJ3CWFXPSJN5BL2FQUTLNHURJRFBUT3XAD5CI6H5YYA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385905035 (396h52m9.15654637s) [noise=true , address=45.91.95.2:4008 , piece_id=WGWH2DZO6DGMDZBAANIDLWHR6MOCHHUZQREZAZZGGS4SE3ILP74A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1WTvNJu1X2JaZAT9x58hxgmx849MyJWXWVsbwe8SfQexjdnJAF, address:45.91.95.2:4008): protocol: write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection; write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection Jul 9 22:02:34.385937026 (396h52m9.156578421s) [seg_pos={0 0}, upload=317505]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 22:02:34.637872527 (396h52m9.408513862s) [upload=317505]: Done uploading segments... Jul 9 22:02:34.637897874 (396h52m9.40853921s) [upload=317505]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 38.928435887s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.237419607s FAIL (2025-07-09 22:08:31 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 6.237448772s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X.X....] duration: 6.075667221s FAIL (2025-07-09 22:08:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.07568782s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [..X...............X..X....X........X.........X..............] duration: 23.188707283s FAIL (2025-07-09 22:08:01 UTC) orbiter warning: context deadline exceeded Duration: 23.188727853s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [X...............X..X.X...X...X.......X............X.X.......] duration: 27.480568989s FAIL (2025-07-09 22:07:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 17505]: Done starting upload Jul 9 22:01:35.031454414 (396h51m9.802095779s) [upload=317505]: Uploading segments... Jul 9 22:01:35.031760129 (396h51m9.802401504s) [upload=317505]: Got next segment. Inline: false Jul 9 22:01:35.031826716 (396h51m9.802468081s) [upload=317505]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:01:35.031834370 (396h51m9.802475725s) [upload=317505]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:01:35.482978326 (396h51m10.253619662s) [seg_pos={0 0}, upload=317505]: Begin upload segment... Jul 9 22:01:35.483027780 (396h51m10.253669115s) [seg_pos={0 0}, upload=317505]: Done begin upload segment. Jul 9 22:01:35.483053139 (396h51m10.253694474s) [noise=true , address=77.90.31.16:4741 , piece_id=XGSTR3METENXDLWEX5V65WJJPVLYCTF5FNN6OWOLNQ35EMYF4YBQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483088858 (396h51m10.253730193s) [noise=true , address=104.36.82.17:4340 , piece_id=7SQQWWMV5Z6JVSOQKHJXCJAECAPNYVSXTNCWH3I2JLF6GEXAUWHQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483512137 (396h51m10.254153472s) [noise=true , address=79.127.219.38:4013 , piece_id=IQPTM37ESAZZJELY4RLPBLBMEFPFYXP63SDVK47P7Y3HN3JKIYEQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483526023 (396h51m10.254167359s) [noise=true , address=45.91.95.3:4018 , piece_id=WEMOWEK3L6A3GLWZBG4BN4JTP6PWPYZ5ATAODXEL6AAJHE2XA7AA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483777826 (396h51m10.254419161s) [noise=true , address=104.36.82.38:4709 , piece_id=CB52JNMPPONRIHQXMBCL4NVC23ONSTA2T4ZVYEU67A3VQZXMMLWA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483903696 (396h51m10.254545031s) [noise=true , address=77.90.31.17:4789 , piece_id=XS44AT7VLC7G6VVVV4HVZHNT7YTYWLAC2EOPDPDSUQ6BNMXQL2EA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484036110 (396h51m10.254677435s) [noise=true , address=45.91.95.5:4033 , piece_id=GOBAVQTDDZOWQHJGUVIKCRNPMEC7WC5GW5BYEQVAYDJKQK7KXAIQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484318500 (396h51m10.254959835s) [noise=true , address=77.90.31.10:4547 , piece_id=GX2EF5YML4GXZ2GH5J54YBV7CGE5JKMC63MECWAMDCZYUA6QUE5A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484411849 (396h51m10.255053184s) [noise=true , address=77.90.31.14:4674 , piece_id=6HVO3E5TKEL5JSCA3AKOS3F6NCPD34L3ACAJGCBKDXMJQNBK2Y2Q, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484600008 (396h51m10.255241343s) [noise=true , address=77.90.31.13:4638 , piece_id=OLHXLVAS74AXSME7T242FXYDRNY3TAASGFWRZG2DQ7Y7TY45M6IQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484754364 (396h51m10.255395699s) [noise=true , address=77.90.31.18:4834 , piece_id=7BABB2K6Z5ROMH7GL4ZS3NMMTCMMKGAK3YQRW45BOQ2YSLVZ6JQQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484809600 (396h51m10.255450935s) [noise=true , address=45.91.95.2:4008 , piece_id=WGWH2DZO6DGMDZBAANIDLWHR6MOCHHUZQREZAZZGGS4SE3ILP74A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485011466 (396h51m10.255652791s) [noise=true , address=45.91.95.4:4028 , piece_id=LNOPV7BHCMO3EAEXSYHXMHR4GJ3277WDWMFZMBE6DAP32RV2G26Q, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485075679 (396h51m10.255717014s) [noise=true , address=79.127.220.98:4416 , piece_id=JMUXH2XXYATER2XYXWQMNOWSMVYVSXKCYGH5MHRIZ2XU2T4WKQZQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485387596 (396h51m10.256028931s) [noise=true , address=77.90.31.21:4912 , piece_id=5ILPD7GX2AGAQXJLQQ5MLKVQOB6I5Q37RQV33EGGSUZIS57WIY2A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485694533 (396h51m10.256335868s) [noise=true , address=104.36.82.60:4339 , piece_id=VFM6GGQB3WJ3CWFXPSJN5BL2FQUTLNHURJRFBUT3XAD5CI6H5YYA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.486011620 (396h51m10.256652955s) [noise=true , address=104.36.82.36:4628 , piece_id=WW4Z3U3GIMLRET2K2KJ4D255ROXIQH5DZBZOJQMGU62YEXKZ6G3A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.486306593 (396h51m10.256947928s) [noise=true , address=104.36.82.44:4875 , piece_id=NPZIJRT65NIYURDZOEQVZW5EJPCDGPL4RVKRQJ4EURFQDJXQZJMA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:36.891865578 (396h51m11.662506913s) [upload=317505]: Next returned nil segment Jul 9 22:01:36.891886257 (396h51m11.662527582s) [upload=317505]: Waiting for error group managing segments... Jul 9 22:01:37.305485395 (396h51m12.07612673s) [noise=true , address=79.127.220.98:4416 , piece_id=JMUXH2XXYATER2XYXWQMNOWSMVYVSXKCYGH5MHRIZ2XU2T4WKQZQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.473014492 (396h51m12.243655827s) [noise=true , address=79.127.219.38:4013 , piece_id=IQPTM37ESAZZJELY4RLPBLBMEFPFYXP63SDVK47P7Y3HN3JKIYEQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.476121286 (396h51m12.246762621s) [noise=true , address=77.90.31.10:4547 , piece_id=GX2EF5YML4GXZ2GH5J54YBV7CGE5JKMC63MECWAMDCZYUA6QUE5A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.486869289 (396h51m12.257510624s) [noise=true , address=77.90.31.17:4789 , piece_id=XS44AT7VLC7G6VVVV4HVZHNT7YTYWLAC2EOPDPDSUQ6BNMXQL2EA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.491485340 (396h51m12.262126695s) [noise=true , address=77.90.31.18:4834 , piece_id=7BABB2K6Z5ROMH7GL4ZS3NMMTCMMKGAK3YQRW45BOQ2YSLVZ6JQQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.491802476 (396h51m12.262443811s) [noise=true , address=77.90.31.16:4741 , piece_id=XGSTR3METENXDLWEX5V65WJJPVLYCTF5FNN6OWOLNQ35EMYF4YBQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.492708459 (396h51m12.263349785s) [noise=true , address=77.90.31.21:4912 , piece_id=5ILPD7GX2AGAQXJLQQ5MLKVQOB6I5Q37RQV33EGGSUZIS57WIY2A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.495223731 (396h51m12.265865087s) [noise=true , address=77.90.31.14:4674 , piece_id=6HVO3E5TKEL5JSCA3AKOS3F6NCPD34L3ACAJGCBKDXMJQNBK2Y2Q, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.496454086 (396h51m12.267095441s) [noise=true , address=77.90.31.13:4638 , piece_id=OLHXLVAS74AXSME7T242FXYDRNY3TAASGFWRZG2DQ7Y7TY45M6IQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:02.845874616 (396h51m37.616515951s) [noise=true , address=45.91.95.4:4028 , piece_id=LNOPV7BHCMO3EAEXSYHXMHR4GJ3277WDWMFZMBE6DAP32RV2G26Q, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:08.932651945 (396h51m43.70329328s) [noise=true , address=104.36.82.44:4875 , piece_id=NPZIJRT65NIYURDZOEQVZW5EJPCDGPL4RVKRQJ4EURFQDJXQZJMA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:17.144956582 (396h51m51.915597917s) [noise=true , address=104.36.82.17:4340 , piece_id=7SQQWWMV5Z6JVSOQKHJXCJAECAPNYVSXTNCWH3I2JLF6GEXAUWHQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:33.266596631 (396h52m8.037237966s) [noise=true , address=45.91.95.5:4033 , piece_id=GOBAVQTDDZOWQHJGUVIKCRNPMEC7WC5GW5BYEQVAYDJKQK7KXAIQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:34.385712938 (396h52m9.156354273s) [noise=true , address=104.36.82.36:4628 , piece_id=WW4Z3U3GIMLRET2K2KJ4D255ROXIQH5DZBZOJQMGU62YEXKZ6G3A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385742234 (396h52m9.156383569s) [noise=true , address=104.36.82.38:4709 , piece_id=CB52JNMPPONRIHQXMBCL4NVC23ONSTA2T4ZVYEU67A3VQZXMMLWA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1Ad77dgyCKuYL8o9Ac1WrMfk7pG1mgkakcVXwZa2QHD6KG5sdH, address:104.36.82.38:4709): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385815304 (396h52m9.156456639s) [noise=true , address=45.91.95.3:4018 , piece_id=WEMOWEK3L6A3GLWZBG4BN4JTP6PWPYZ5ATAODXEL6AAJHE2XA7AA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:12qwhpXGKbp4BZ5xTgzYZyw297QcYcrTWo3zUeXVecXsfRqxB6c, address:45.91.95.3:4018): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385853136 (396h52m9.156494471s) [noise=true , address=104.36.82.60:4339 , piece_id=VFM6GGQB3WJ3CWFXPSJN5BL2FQUTLNHURJRFBUT3XAD5CI6H5YYA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385905035 (396h52m9.15654637s) [noise=true , address=45.91.95.2:4008 , piece_id=WGWH2DZO6DGMDZBAANIDLWHR6MOCHHUZQREZAZZGGS4SE3ILP74A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1WTvNJu1X2JaZAT9x58hxgmx849MyJWXWVsbwe8SfQexjdnJAF, address:45.91.95.2:4008): protocol: write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection; write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection Jul 9 22:02:34.385937026 (396h52m9.156578421s) [seg_pos={0 0}, upload=317505]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 22:02:34.637872527 (396h52m9.408513862s) [upload=317505]: Done uploading segments... Jul 9 22:02:34.637897874 (396h52m9.40853921s) [upload=317505]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 27.480594095s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 4.837571677s FAIL (2025-07-09 22:07:38 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.837596835s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [...X...............X..X....X........X.........X.............] duration: 20.469985939s FAIL (2025-07-09 22:07:13 UTC) orbiter warning: context deadline exceeded Duration: 20.470011037s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X.X...] duration: 5.927726415s FAIL (2025-07-09 22:07:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.927814233s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.X...............X..X.X...X...X.......X............X.X......] duration: 26.848399626s FAIL (2025-07-09 22:06:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 17505]: Done starting upload Jul 9 22:01:35.031454414 (396h51m9.802095779s) [upload=317505]: Uploading segments... Jul 9 22:01:35.031760129 (396h51m9.802401504s) [upload=317505]: Got next segment. Inline: false Jul 9 22:01:35.031826716 (396h51m9.802468081s) [upload=317505]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:01:35.031834370 (396h51m9.802475725s) [upload=317505]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:01:35.482978326 (396h51m10.253619662s) [seg_pos={0 0}, upload=317505]: Begin upload segment... Jul 9 22:01:35.483027780 (396h51m10.253669115s) [seg_pos={0 0}, upload=317505]: Done begin upload segment. Jul 9 22:01:35.483053139 (396h51m10.253694474s) [noise=true , address=77.90.31.16:4741 , piece_id=XGSTR3METENXDLWEX5V65WJJPVLYCTF5FNN6OWOLNQ35EMYF4YBQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483088858 (396h51m10.253730193s) [noise=true , address=104.36.82.17:4340 , piece_id=7SQQWWMV5Z6JVSOQKHJXCJAECAPNYVSXTNCWH3I2JLF6GEXAUWHQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483512137 (396h51m10.254153472s) [noise=true , address=79.127.219.38:4013 , piece_id=IQPTM37ESAZZJELY4RLPBLBMEFPFYXP63SDVK47P7Y3HN3JKIYEQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483526023 (396h51m10.254167359s) [noise=true , address=45.91.95.3:4018 , piece_id=WEMOWEK3L6A3GLWZBG4BN4JTP6PWPYZ5ATAODXEL6AAJHE2XA7AA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483777826 (396h51m10.254419161s) [noise=true , address=104.36.82.38:4709 , piece_id=CB52JNMPPONRIHQXMBCL4NVC23ONSTA2T4ZVYEU67A3VQZXMMLWA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483903696 (396h51m10.254545031s) [noise=true , address=77.90.31.17:4789 , piece_id=XS44AT7VLC7G6VVVV4HVZHNT7YTYWLAC2EOPDPDSUQ6BNMXQL2EA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484036110 (396h51m10.254677435s) [noise=true , address=45.91.95.5:4033 , piece_id=GOBAVQTDDZOWQHJGUVIKCRNPMEC7WC5GW5BYEQVAYDJKQK7KXAIQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484318500 (396h51m10.254959835s) [noise=true , address=77.90.31.10:4547 , piece_id=GX2EF5YML4GXZ2GH5J54YBV7CGE5JKMC63MECWAMDCZYUA6QUE5A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484411849 (396h51m10.255053184s) [noise=true , address=77.90.31.14:4674 , piece_id=6HVO3E5TKEL5JSCA3AKOS3F6NCPD34L3ACAJGCBKDXMJQNBK2Y2Q, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484600008 (396h51m10.255241343s) [noise=true , address=77.90.31.13:4638 , piece_id=OLHXLVAS74AXSME7T242FXYDRNY3TAASGFWRZG2DQ7Y7TY45M6IQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484754364 (396h51m10.255395699s) [noise=true , address=77.90.31.18:4834 , piece_id=7BABB2K6Z5ROMH7GL4ZS3NMMTCMMKGAK3YQRW45BOQ2YSLVZ6JQQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484809600 (396h51m10.255450935s) [noise=true , address=45.91.95.2:4008 , piece_id=WGWH2DZO6DGMDZBAANIDLWHR6MOCHHUZQREZAZZGGS4SE3ILP74A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485011466 (396h51m10.255652791s) [noise=true , address=45.91.95.4:4028 , piece_id=LNOPV7BHCMO3EAEXSYHXMHR4GJ3277WDWMFZMBE6DAP32RV2G26Q, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485075679 (396h51m10.255717014s) [noise=true , address=79.127.220.98:4416 , piece_id=JMUXH2XXYATER2XYXWQMNOWSMVYVSXKCYGH5MHRIZ2XU2T4WKQZQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485387596 (396h51m10.256028931s) [noise=true , address=77.90.31.21:4912 , piece_id=5ILPD7GX2AGAQXJLQQ5MLKVQOB6I5Q37RQV33EGGSUZIS57WIY2A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485694533 (396h51m10.256335868s) [noise=true , address=104.36.82.60:4339 , piece_id=VFM6GGQB3WJ3CWFXPSJN5BL2FQUTLNHURJRFBUT3XAD5CI6H5YYA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.486011620 (396h51m10.256652955s) [noise=true , address=104.36.82.36:4628 , piece_id=WW4Z3U3GIMLRET2K2KJ4D255ROXIQH5DZBZOJQMGU62YEXKZ6G3A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.486306593 (396h51m10.256947928s) [noise=true , address=104.36.82.44:4875 , piece_id=NPZIJRT65NIYURDZOEQVZW5EJPCDGPL4RVKRQJ4EURFQDJXQZJMA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:36.891865578 (396h51m11.662506913s) [upload=317505]: Next returned nil segment Jul 9 22:01:36.891886257 (396h51m11.662527582s) [upload=317505]: Waiting for error group managing segments... Jul 9 22:01:37.305485395 (396h51m12.07612673s) [noise=true , address=79.127.220.98:4416 , piece_id=JMUXH2XXYATER2XYXWQMNOWSMVYVSXKCYGH5MHRIZ2XU2T4WKQZQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.473014492 (396h51m12.243655827s) [noise=true , address=79.127.219.38:4013 , piece_id=IQPTM37ESAZZJELY4RLPBLBMEFPFYXP63SDVK47P7Y3HN3JKIYEQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.476121286 (396h51m12.246762621s) [noise=true , address=77.90.31.10:4547 , piece_id=GX2EF5YML4GXZ2GH5J54YBV7CGE5JKMC63MECWAMDCZYUA6QUE5A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.486869289 (396h51m12.257510624s) [noise=true , address=77.90.31.17:4789 , piece_id=XS44AT7VLC7G6VVVV4HVZHNT7YTYWLAC2EOPDPDSUQ6BNMXQL2EA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.491485340 (396h51m12.262126695s) [noise=true , address=77.90.31.18:4834 , piece_id=7BABB2K6Z5ROMH7GL4ZS3NMMTCMMKGAK3YQRW45BOQ2YSLVZ6JQQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.491802476 (396h51m12.262443811s) [noise=true , address=77.90.31.16:4741 , piece_id=XGSTR3METENXDLWEX5V65WJJPVLYCTF5FNN6OWOLNQ35EMYF4YBQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.492708459 (396h51m12.263349785s) [noise=true , address=77.90.31.21:4912 , piece_id=5ILPD7GX2AGAQXJLQQ5MLKVQOB6I5Q37RQV33EGGSUZIS57WIY2A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.495223731 (396h51m12.265865087s) [noise=true , address=77.90.31.14:4674 , piece_id=6HVO3E5TKEL5JSCA3AKOS3F6NCPD34L3ACAJGCBKDXMJQNBK2Y2Q, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.496454086 (396h51m12.267095441s) [noise=true , address=77.90.31.13:4638 , piece_id=OLHXLVAS74AXSME7T242FXYDRNY3TAASGFWRZG2DQ7Y7TY45M6IQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:02.845874616 (396h51m37.616515951s) [noise=true , address=45.91.95.4:4028 , piece_id=LNOPV7BHCMO3EAEXSYHXMHR4GJ3277WDWMFZMBE6DAP32RV2G26Q, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:08.932651945 (396h51m43.70329328s) [noise=true , address=104.36.82.44:4875 , piece_id=NPZIJRT65NIYURDZOEQVZW5EJPCDGPL4RVKRQJ4EURFQDJXQZJMA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:17.144956582 (396h51m51.915597917s) [noise=true , address=104.36.82.17:4340 , piece_id=7SQQWWMV5Z6JVSOQKHJXCJAECAPNYVSXTNCWH3I2JLF6GEXAUWHQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:33.266596631 (396h52m8.037237966s) [noise=true , address=45.91.95.5:4033 , piece_id=GOBAVQTDDZOWQHJGUVIKCRNPMEC7WC5GW5BYEQVAYDJKQK7KXAIQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:34.385712938 (396h52m9.156354273s) [noise=true , address=104.36.82.36:4628 , piece_id=WW4Z3U3GIMLRET2K2KJ4D255ROXIQH5DZBZOJQMGU62YEXKZ6G3A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385742234 (396h52m9.156383569s) [noise=true , address=104.36.82.38:4709 , piece_id=CB52JNMPPONRIHQXMBCL4NVC23ONSTA2T4ZVYEU67A3VQZXMMLWA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1Ad77dgyCKuYL8o9Ac1WrMfk7pG1mgkakcVXwZa2QHD6KG5sdH, address:104.36.82.38:4709): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385815304 (396h52m9.156456639s) [noise=true , address=45.91.95.3:4018 , piece_id=WEMOWEK3L6A3GLWZBG4BN4JTP6PWPYZ5ATAODXEL6AAJHE2XA7AA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:12qwhpXGKbp4BZ5xTgzYZyw297QcYcrTWo3zUeXVecXsfRqxB6c, address:45.91.95.3:4018): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385853136 (396h52m9.156494471s) [noise=true , address=104.36.82.60:4339 , piece_id=VFM6GGQB3WJ3CWFXPSJN5BL2FQUTLNHURJRFBUT3XAD5CI6H5YYA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385905035 (396h52m9.15654637s) [noise=true , address=45.91.95.2:4008 , piece_id=WGWH2DZO6DGMDZBAANIDLWHR6MOCHHUZQREZAZZGGS4SE3ILP74A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1WTvNJu1X2JaZAT9x58hxgmx849MyJWXWVsbwe8SfQexjdnJAF, address:45.91.95.2:4008): protocol: write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection; write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection Jul 9 22:02:34.385937026 (396h52m9.156578421s) [seg_pos={0 0}, upload=317505]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 22:02:34.637872527 (396h52m9.408513862s) [upload=317505]: Done uploading segments... Jul 9 22:02:34.637897874 (396h52m9.40853921s) [upload=317505]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 26.848426506s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.06343598s FAIL (2025-07-09 22:06:39 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 5.063463784s core uplink eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X.X..] duration: 11.848655892s FAIL (2025-07-09 22:06:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 11.848677552s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [....X...............X..X....X........X.........X............] duration: 33.981930822s FAIL (2025-07-09 22:06:20 UTC) orbiter warning: context deadline exceeded Duration: 33.981942664s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................X.X.] duration: 6.121441324s FAIL (2025-07-09 22:06:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.121458296s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [..X...............X..X.X...X...X.......X............X.X.....] duration: 25.672113495s FAIL (2025-07-09 22:05:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 17505]: Done starting upload Jul 9 22:01:35.031454414 (396h51m9.802095779s) [upload=317505]: Uploading segments... Jul 9 22:01:35.031760129 (396h51m9.802401504s) [upload=317505]: Got next segment. Inline: false Jul 9 22:01:35.031826716 (396h51m9.802468081s) [upload=317505]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:01:35.031834370 (396h51m9.802475725s) [upload=317505]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:01:35.482978326 (396h51m10.253619662s) [seg_pos={0 0}, upload=317505]: Begin upload segment... Jul 9 22:01:35.483027780 (396h51m10.253669115s) [seg_pos={0 0}, upload=317505]: Done begin upload segment. Jul 9 22:01:35.483053139 (396h51m10.253694474s) [noise=true , address=77.90.31.16:4741 , piece_id=XGSTR3METENXDLWEX5V65WJJPVLYCTF5FNN6OWOLNQ35EMYF4YBQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483088858 (396h51m10.253730193s) [noise=true , address=104.36.82.17:4340 , piece_id=7SQQWWMV5Z6JVSOQKHJXCJAECAPNYVSXTNCWH3I2JLF6GEXAUWHQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483512137 (396h51m10.254153472s) [noise=true , address=79.127.219.38:4013 , piece_id=IQPTM37ESAZZJELY4RLPBLBMEFPFYXP63SDVK47P7Y3HN3JKIYEQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483526023 (396h51m10.254167359s) [noise=true , address=45.91.95.3:4018 , piece_id=WEMOWEK3L6A3GLWZBG4BN4JTP6PWPYZ5ATAODXEL6AAJHE2XA7AA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483777826 (396h51m10.254419161s) [noise=true , address=104.36.82.38:4709 , piece_id=CB52JNMPPONRIHQXMBCL4NVC23ONSTA2T4ZVYEU67A3VQZXMMLWA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483903696 (396h51m10.254545031s) [noise=true , address=77.90.31.17:4789 , piece_id=XS44AT7VLC7G6VVVV4HVZHNT7YTYWLAC2EOPDPDSUQ6BNMXQL2EA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484036110 (396h51m10.254677435s) [noise=true , address=45.91.95.5:4033 , piece_id=GOBAVQTDDZOWQHJGUVIKCRNPMEC7WC5GW5BYEQVAYDJKQK7KXAIQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484318500 (396h51m10.254959835s) [noise=true , address=77.90.31.10:4547 , piece_id=GX2EF5YML4GXZ2GH5J54YBV7CGE5JKMC63MECWAMDCZYUA6QUE5A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484411849 (396h51m10.255053184s) [noise=true , address=77.90.31.14:4674 , piece_id=6HVO3E5TKEL5JSCA3AKOS3F6NCPD34L3ACAJGCBKDXMJQNBK2Y2Q, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484600008 (396h51m10.255241343s) [noise=true , address=77.90.31.13:4638 , piece_id=OLHXLVAS74AXSME7T242FXYDRNY3TAASGFWRZG2DQ7Y7TY45M6IQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484754364 (396h51m10.255395699s) [noise=true , address=77.90.31.18:4834 , piece_id=7BABB2K6Z5ROMH7GL4ZS3NMMTCMMKGAK3YQRW45BOQ2YSLVZ6JQQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484809600 (396h51m10.255450935s) [noise=true , address=45.91.95.2:4008 , piece_id=WGWH2DZO6DGMDZBAANIDLWHR6MOCHHUZQREZAZZGGS4SE3ILP74A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485011466 (396h51m10.255652791s) [noise=true , address=45.91.95.4:4028 , piece_id=LNOPV7BHCMO3EAEXSYHXMHR4GJ3277WDWMFZMBE6DAP32RV2G26Q, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485075679 (396h51m10.255717014s) [noise=true , address=79.127.220.98:4416 , piece_id=JMUXH2XXYATER2XYXWQMNOWSMVYVSXKCYGH5MHRIZ2XU2T4WKQZQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485387596 (396h51m10.256028931s) [noise=true , address=77.90.31.21:4912 , piece_id=5ILPD7GX2AGAQXJLQQ5MLKVQOB6I5Q37RQV33EGGSUZIS57WIY2A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485694533 (396h51m10.256335868s) [noise=true , address=104.36.82.60:4339 , piece_id=VFM6GGQB3WJ3CWFXPSJN5BL2FQUTLNHURJRFBUT3XAD5CI6H5YYA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.486011620 (396h51m10.256652955s) [noise=true , address=104.36.82.36:4628 , piece_id=WW4Z3U3GIMLRET2K2KJ4D255ROXIQH5DZBZOJQMGU62YEXKZ6G3A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.486306593 (396h51m10.256947928s) [noise=true , address=104.36.82.44:4875 , piece_id=NPZIJRT65NIYURDZOEQVZW5EJPCDGPL4RVKRQJ4EURFQDJXQZJMA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:36.891865578 (396h51m11.662506913s) [upload=317505]: Next returned nil segment Jul 9 22:01:36.891886257 (396h51m11.662527582s) [upload=317505]: Waiting for error group managing segments... Jul 9 22:01:37.305485395 (396h51m12.07612673s) [noise=true , address=79.127.220.98:4416 , piece_id=JMUXH2XXYATER2XYXWQMNOWSMVYVSXKCYGH5MHRIZ2XU2T4WKQZQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.473014492 (396h51m12.243655827s) [noise=true , address=79.127.219.38:4013 , piece_id=IQPTM37ESAZZJELY4RLPBLBMEFPFYXP63SDVK47P7Y3HN3JKIYEQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.476121286 (396h51m12.246762621s) [noise=true , address=77.90.31.10:4547 , piece_id=GX2EF5YML4GXZ2GH5J54YBV7CGE5JKMC63MECWAMDCZYUA6QUE5A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.486869289 (396h51m12.257510624s) [noise=true , address=77.90.31.17:4789 , piece_id=XS44AT7VLC7G6VVVV4HVZHNT7YTYWLAC2EOPDPDSUQ6BNMXQL2EA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.491485340 (396h51m12.262126695s) [noise=true , address=77.90.31.18:4834 , piece_id=7BABB2K6Z5ROMH7GL4ZS3NMMTCMMKGAK3YQRW45BOQ2YSLVZ6JQQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.491802476 (396h51m12.262443811s) [noise=true , address=77.90.31.16:4741 , piece_id=XGSTR3METENXDLWEX5V65WJJPVLYCTF5FNN6OWOLNQ35EMYF4YBQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.492708459 (396h51m12.263349785s) [noise=true , address=77.90.31.21:4912 , piece_id=5ILPD7GX2AGAQXJLQQ5MLKVQOB6I5Q37RQV33EGGSUZIS57WIY2A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.495223731 (396h51m12.265865087s) [noise=true , address=77.90.31.14:4674 , piece_id=6HVO3E5TKEL5JSCA3AKOS3F6NCPD34L3ACAJGCBKDXMJQNBK2Y2Q, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.496454086 (396h51m12.267095441s) [noise=true , address=77.90.31.13:4638 , piece_id=OLHXLVAS74AXSME7T242FXYDRNY3TAASGFWRZG2DQ7Y7TY45M6IQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:02.845874616 (396h51m37.616515951s) [noise=true , address=45.91.95.4:4028 , piece_id=LNOPV7BHCMO3EAEXSYHXMHR4GJ3277WDWMFZMBE6DAP32RV2G26Q, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:08.932651945 (396h51m43.70329328s) [noise=true , address=104.36.82.44:4875 , piece_id=NPZIJRT65NIYURDZOEQVZW5EJPCDGPL4RVKRQJ4EURFQDJXQZJMA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:17.144956582 (396h51m51.915597917s) [noise=true , address=104.36.82.17:4340 , piece_id=7SQQWWMV5Z6JVSOQKHJXCJAECAPNYVSXTNCWH3I2JLF6GEXAUWHQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:33.266596631 (396h52m8.037237966s) [noise=true , address=45.91.95.5:4033 , piece_id=GOBAVQTDDZOWQHJGUVIKCRNPMEC7WC5GW5BYEQVAYDJKQK7KXAIQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:34.385712938 (396h52m9.156354273s) [noise=true , address=104.36.82.36:4628 , piece_id=WW4Z3U3GIMLRET2K2KJ4D255ROXIQH5DZBZOJQMGU62YEXKZ6G3A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385742234 (396h52m9.156383569s) [noise=true , address=104.36.82.38:4709 , piece_id=CB52JNMPPONRIHQXMBCL4NVC23ONSTA2T4ZVYEU67A3VQZXMMLWA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1Ad77dgyCKuYL8o9Ac1WrMfk7pG1mgkakcVXwZa2QHD6KG5sdH, address:104.36.82.38:4709): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385815304 (396h52m9.156456639s) [noise=true , address=45.91.95.3:4018 , piece_id=WEMOWEK3L6A3GLWZBG4BN4JTP6PWPYZ5ATAODXEL6AAJHE2XA7AA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:12qwhpXGKbp4BZ5xTgzYZyw297QcYcrTWo3zUeXVecXsfRqxB6c, address:45.91.95.3:4018): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385853136 (396h52m9.156494471s) [noise=true , address=104.36.82.60:4339 , piece_id=VFM6GGQB3WJ3CWFXPSJN5BL2FQUTLNHURJRFBUT3XAD5CI6H5YYA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385905035 (396h52m9.15654637s) [noise=true , address=45.91.95.2:4008 , piece_id=WGWH2DZO6DGMDZBAANIDLWHR6MOCHHUZQREZAZZGGS4SE3ILP74A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1WTvNJu1X2JaZAT9x58hxgmx849MyJWXWVsbwe8SfQexjdnJAF, address:45.91.95.2:4008): protocol: write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection; write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection Jul 9 22:02:34.385937026 (396h52m9.156578421s) [seg_pos={0 0}, upload=317505]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 22:02:34.637872527 (396h52m9.408513862s) [upload=317505]: Done uploading segments... Jul 9 22:02:34.637897874 (396h52m9.40853921s) [upload=317505]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 25.672139935s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.....X...............X..X....X........X.........X...........] duration: 34.70248793s FAIL (2025-07-09 22:05:46 UTC) orbiter warning: context deadline exceeded Duration: 34.702507556s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................X.X] duration: 30.000921972s FAIL (2025-07-09 22:05:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000940558s edge gateway us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.439088277s FAIL (2025-07-09 22:05:38 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 5.439118306s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [...X...............X..X.X...X...X.......X............X.X....] duration: 49.453721138s FAIL (2025-07-09 22:05:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 17505]: Done starting upload Jul 9 22:01:35.031454414 (396h51m9.802095779s) [upload=317505]: Uploading segments... Jul 9 22:01:35.031760129 (396h51m9.802401504s) [upload=317505]: Got next segment. Inline: false Jul 9 22:01:35.031826716 (396h51m9.802468081s) [upload=317505]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:01:35.031834370 (396h51m9.802475725s) [upload=317505]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:01:35.482978326 (396h51m10.253619662s) [seg_pos={0 0}, upload=317505]: Begin upload segment... Jul 9 22:01:35.483027780 (396h51m10.253669115s) [seg_pos={0 0}, upload=317505]: Done begin upload segment. Jul 9 22:01:35.483053139 (396h51m10.253694474s) [noise=true , address=77.90.31.16:4741 , piece_id=XGSTR3METENXDLWEX5V65WJJPVLYCTF5FNN6OWOLNQ35EMYF4YBQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483088858 (396h51m10.253730193s) [noise=true , address=104.36.82.17:4340 , piece_id=7SQQWWMV5Z6JVSOQKHJXCJAECAPNYVSXTNCWH3I2JLF6GEXAUWHQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483512137 (396h51m10.254153472s) [noise=true , address=79.127.219.38:4013 , piece_id=IQPTM37ESAZZJELY4RLPBLBMEFPFYXP63SDVK47P7Y3HN3JKIYEQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483526023 (396h51m10.254167359s) [noise=true , address=45.91.95.3:4018 , piece_id=WEMOWEK3L6A3GLWZBG4BN4JTP6PWPYZ5ATAODXEL6AAJHE2XA7AA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483777826 (396h51m10.254419161s) [noise=true , address=104.36.82.38:4709 , piece_id=CB52JNMPPONRIHQXMBCL4NVC23ONSTA2T4ZVYEU67A3VQZXMMLWA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483903696 (396h51m10.254545031s) [noise=true , address=77.90.31.17:4789 , piece_id=XS44AT7VLC7G6VVVV4HVZHNT7YTYWLAC2EOPDPDSUQ6BNMXQL2EA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484036110 (396h51m10.254677435s) [noise=true , address=45.91.95.5:4033 , piece_id=GOBAVQTDDZOWQHJGUVIKCRNPMEC7WC5GW5BYEQVAYDJKQK7KXAIQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484318500 (396h51m10.254959835s) [noise=true , address=77.90.31.10:4547 , piece_id=GX2EF5YML4GXZ2GH5J54YBV7CGE5JKMC63MECWAMDCZYUA6QUE5A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484411849 (396h51m10.255053184s) [noise=true , address=77.90.31.14:4674 , piece_id=6HVO3E5TKEL5JSCA3AKOS3F6NCPD34L3ACAJGCBKDXMJQNBK2Y2Q, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484600008 (396h51m10.255241343s) [noise=true , address=77.90.31.13:4638 , piece_id=OLHXLVAS74AXSME7T242FXYDRNY3TAASGFWRZG2DQ7Y7TY45M6IQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484754364 (396h51m10.255395699s) [noise=true , address=77.90.31.18:4834 , piece_id=7BABB2K6Z5ROMH7GL4ZS3NMMTCMMKGAK3YQRW45BOQ2YSLVZ6JQQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484809600 (396h51m10.255450935s) [noise=true , address=45.91.95.2:4008 , piece_id=WGWH2DZO6DGMDZBAANIDLWHR6MOCHHUZQREZAZZGGS4SE3ILP74A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485011466 (396h51m10.255652791s) [noise=true , address=45.91.95.4:4028 , piece_id=LNOPV7BHCMO3EAEXSYHXMHR4GJ3277WDWMFZMBE6DAP32RV2G26Q, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485075679 (396h51m10.255717014s) [noise=true , address=79.127.220.98:4416 , piece_id=JMUXH2XXYATER2XYXWQMNOWSMVYVSXKCYGH5MHRIZ2XU2T4WKQZQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485387596 (396h51m10.256028931s) [noise=true , address=77.90.31.21:4912 , piece_id=5ILPD7GX2AGAQXJLQQ5MLKVQOB6I5Q37RQV33EGGSUZIS57WIY2A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485694533 (396h51m10.256335868s) [noise=true , address=104.36.82.60:4339 , piece_id=VFM6GGQB3WJ3CWFXPSJN5BL2FQUTLNHURJRFBUT3XAD5CI6H5YYA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.486011620 (396h51m10.256652955s) [noise=true , address=104.36.82.36:4628 , piece_id=WW4Z3U3GIMLRET2K2KJ4D255ROXIQH5DZBZOJQMGU62YEXKZ6G3A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.486306593 (396h51m10.256947928s) [noise=true , address=104.36.82.44:4875 , piece_id=NPZIJRT65NIYURDZOEQVZW5EJPCDGPL4RVKRQJ4EURFQDJXQZJMA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:36.891865578 (396h51m11.662506913s) [upload=317505]: Next returned nil segment Jul 9 22:01:36.891886257 (396h51m11.662527582s) [upload=317505]: Waiting for error group managing segments... Jul 9 22:01:37.305485395 (396h51m12.07612673s) [noise=true , address=79.127.220.98:4416 , piece_id=JMUXH2XXYATER2XYXWQMNOWSMVYVSXKCYGH5MHRIZ2XU2T4WKQZQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.473014492 (396h51m12.243655827s) [noise=true , address=79.127.219.38:4013 , piece_id=IQPTM37ESAZZJELY4RLPBLBMEFPFYXP63SDVK47P7Y3HN3JKIYEQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.476121286 (396h51m12.246762621s) [noise=true , address=77.90.31.10:4547 , piece_id=GX2EF5YML4GXZ2GH5J54YBV7CGE5JKMC63MECWAMDCZYUA6QUE5A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.486869289 (396h51m12.257510624s) [noise=true , address=77.90.31.17:4789 , piece_id=XS44AT7VLC7G6VVVV4HVZHNT7YTYWLAC2EOPDPDSUQ6BNMXQL2EA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.491485340 (396h51m12.262126695s) [noise=true , address=77.90.31.18:4834 , piece_id=7BABB2K6Z5ROMH7GL4ZS3NMMTCMMKGAK3YQRW45BOQ2YSLVZ6JQQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.491802476 (396h51m12.262443811s) [noise=true , address=77.90.31.16:4741 , piece_id=XGSTR3METENXDLWEX5V65WJJPVLYCTF5FNN6OWOLNQ35EMYF4YBQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.492708459 (396h51m12.263349785s) [noise=true , address=77.90.31.21:4912 , piece_id=5ILPD7GX2AGAQXJLQQ5MLKVQOB6I5Q37RQV33EGGSUZIS57WIY2A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.495223731 (396h51m12.265865087s) [noise=true , address=77.90.31.14:4674 , piece_id=6HVO3E5TKEL5JSCA3AKOS3F6NCPD34L3ACAJGCBKDXMJQNBK2Y2Q, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.496454086 (396h51m12.267095441s) [noise=true , address=77.90.31.13:4638 , piece_id=OLHXLVAS74AXSME7T242FXYDRNY3TAASGFWRZG2DQ7Y7TY45M6IQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:02.845874616 (396h51m37.616515951s) [noise=true , address=45.91.95.4:4028 , piece_id=LNOPV7BHCMO3EAEXSYHXMHR4GJ3277WDWMFZMBE6DAP32RV2G26Q, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:08.932651945 (396h51m43.70329328s) [noise=true , address=104.36.82.44:4875 , piece_id=NPZIJRT65NIYURDZOEQVZW5EJPCDGPL4RVKRQJ4EURFQDJXQZJMA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:17.144956582 (396h51m51.915597917s) [noise=true , address=104.36.82.17:4340 , piece_id=7SQQWWMV5Z6JVSOQKHJXCJAECAPNYVSXTNCWH3I2JLF6GEXAUWHQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:33.266596631 (396h52m8.037237966s) [noise=true , address=45.91.95.5:4033 , piece_id=GOBAVQTDDZOWQHJGUVIKCRNPMEC7WC5GW5BYEQVAYDJKQK7KXAIQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:34.385712938 (396h52m9.156354273s) [noise=true , address=104.36.82.36:4628 , piece_id=WW4Z3U3GIMLRET2K2KJ4D255ROXIQH5DZBZOJQMGU62YEXKZ6G3A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385742234 (396h52m9.156383569s) [noise=true , address=104.36.82.38:4709 , piece_id=CB52JNMPPONRIHQXMBCL4NVC23ONSTA2T4ZVYEU67A3VQZXMMLWA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1Ad77dgyCKuYL8o9Ac1WrMfk7pG1mgkakcVXwZa2QHD6KG5sdH, address:104.36.82.38:4709): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385815304 (396h52m9.156456639s) [noise=true , address=45.91.95.3:4018 , piece_id=WEMOWEK3L6A3GLWZBG4BN4JTP6PWPYZ5ATAODXEL6AAJHE2XA7AA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:12qwhpXGKbp4BZ5xTgzYZyw297QcYcrTWo3zUeXVecXsfRqxB6c, address:45.91.95.3:4018): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385853136 (396h52m9.156494471s) [noise=true , address=104.36.82.60:4339 , piece_id=VFM6GGQB3WJ3CWFXPSJN5BL2FQUTLNHURJRFBUT3XAD5CI6H5YYA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385905035 (396h52m9.15654637s) [noise=true , address=45.91.95.2:4008 , piece_id=WGWH2DZO6DGMDZBAANIDLWHR6MOCHHUZQREZAZZGGS4SE3ILP74A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1WTvNJu1X2JaZAT9x58hxgmx849MyJWXWVsbwe8SfQexjdnJAF, address:45.91.95.2:4008): protocol: write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection; write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection Jul 9 22:02:34.385937026 (396h52m9.156578421s) [seg_pos={0 0}, upload=317505]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 22:02:34.637872527 (396h52m9.408513862s) [upload=317505]: Done uploading segments... Jul 9 22:02:34.637897874 (396h52m9.40853921s) [upload=317505]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 49.453747769s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 4.954891759s FAIL (2025-07-09 22:04:28 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.954919602s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [......X...............X..X....X........X.........X..........] duration: 26.934728841s FAIL (2025-07-09 22:04:25 UTC) orbiter warning: context deadline exceeded Duration: 26.934742168s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 20.684861372s FAIL (2025-07-09 22:04:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.684872302s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [....X...............X..X.X...X...X.......X............X.X...] duration: 20.981107945s FAIL (2025-07-09 22:04:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 17505]: Done starting upload Jul 9 22:01:35.031454414 (396h51m9.802095779s) [upload=317505]: Uploading segments... Jul 9 22:01:35.031760129 (396h51m9.802401504s) [upload=317505]: Got next segment. Inline: false Jul 9 22:01:35.031826716 (396h51m9.802468081s) [upload=317505]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:01:35.031834370 (396h51m9.802475725s) [upload=317505]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:01:35.482978326 (396h51m10.253619662s) [seg_pos={0 0}, upload=317505]: Begin upload segment... Jul 9 22:01:35.483027780 (396h51m10.253669115s) [seg_pos={0 0}, upload=317505]: Done begin upload segment. Jul 9 22:01:35.483053139 (396h51m10.253694474s) [noise=true , address=77.90.31.16:4741 , piece_id=XGSTR3METENXDLWEX5V65WJJPVLYCTF5FNN6OWOLNQ35EMYF4YBQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483088858 (396h51m10.253730193s) [noise=true , address=104.36.82.17:4340 , piece_id=7SQQWWMV5Z6JVSOQKHJXCJAECAPNYVSXTNCWH3I2JLF6GEXAUWHQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483512137 (396h51m10.254153472s) [noise=true , address=79.127.219.38:4013 , piece_id=IQPTM37ESAZZJELY4RLPBLBMEFPFYXP63SDVK47P7Y3HN3JKIYEQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483526023 (396h51m10.254167359s) [noise=true , address=45.91.95.3:4018 , piece_id=WEMOWEK3L6A3GLWZBG4BN4JTP6PWPYZ5ATAODXEL6AAJHE2XA7AA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483777826 (396h51m10.254419161s) [noise=true , address=104.36.82.38:4709 , piece_id=CB52JNMPPONRIHQXMBCL4NVC23ONSTA2T4ZVYEU67A3VQZXMMLWA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483903696 (396h51m10.254545031s) [noise=true , address=77.90.31.17:4789 , piece_id=XS44AT7VLC7G6VVVV4HVZHNT7YTYWLAC2EOPDPDSUQ6BNMXQL2EA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484036110 (396h51m10.254677435s) [noise=true , address=45.91.95.5:4033 , piece_id=GOBAVQTDDZOWQHJGUVIKCRNPMEC7WC5GW5BYEQVAYDJKQK7KXAIQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484318500 (396h51m10.254959835s) [noise=true , address=77.90.31.10:4547 , piece_id=GX2EF5YML4GXZ2GH5J54YBV7CGE5JKMC63MECWAMDCZYUA6QUE5A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484411849 (396h51m10.255053184s) [noise=true , address=77.90.31.14:4674 , piece_id=6HVO3E5TKEL5JSCA3AKOS3F6NCPD34L3ACAJGCBKDXMJQNBK2Y2Q, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484600008 (396h51m10.255241343s) [noise=true , address=77.90.31.13:4638 , piece_id=OLHXLVAS74AXSME7T242FXYDRNY3TAASGFWRZG2DQ7Y7TY45M6IQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484754364 (396h51m10.255395699s) [noise=true , address=77.90.31.18:4834 , piece_id=7BABB2K6Z5ROMH7GL4ZS3NMMTCMMKGAK3YQRW45BOQ2YSLVZ6JQQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484809600 (396h51m10.255450935s) [noise=true , address=45.91.95.2:4008 , piece_id=WGWH2DZO6DGMDZBAANIDLWHR6MOCHHUZQREZAZZGGS4SE3ILP74A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485011466 (396h51m10.255652791s) [noise=true , address=45.91.95.4:4028 , piece_id=LNOPV7BHCMO3EAEXSYHXMHR4GJ3277WDWMFZMBE6DAP32RV2G26Q, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485075679 (396h51m10.255717014s) [noise=true , address=79.127.220.98:4416 , piece_id=JMUXH2XXYATER2XYXWQMNOWSMVYVSXKCYGH5MHRIZ2XU2T4WKQZQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485387596 (396h51m10.256028931s) [noise=true , address=77.90.31.21:4912 , piece_id=5ILPD7GX2AGAQXJLQQ5MLKVQOB6I5Q37RQV33EGGSUZIS57WIY2A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485694533 (396h51m10.256335868s) [noise=true , address=104.36.82.60:4339 , piece_id=VFM6GGQB3WJ3CWFXPSJN5BL2FQUTLNHURJRFBUT3XAD5CI6H5YYA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.486011620 (396h51m10.256652955s) [noise=true , address=104.36.82.36:4628 , piece_id=WW4Z3U3GIMLRET2K2KJ4D255ROXIQH5DZBZOJQMGU62YEXKZ6G3A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.486306593 (396h51m10.256947928s) [noise=true , address=104.36.82.44:4875 , piece_id=NPZIJRT65NIYURDZOEQVZW5EJPCDGPL4RVKRQJ4EURFQDJXQZJMA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:36.891865578 (396h51m11.662506913s) [upload=317505]: Next returned nil segment Jul 9 22:01:36.891886257 (396h51m11.662527582s) [upload=317505]: Waiting for error group managing segments... Jul 9 22:01:37.305485395 (396h51m12.07612673s) [noise=true , address=79.127.220.98:4416 , piece_id=JMUXH2XXYATER2XYXWQMNOWSMVYVSXKCYGH5MHRIZ2XU2T4WKQZQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.473014492 (396h51m12.243655827s) [noise=true , address=79.127.219.38:4013 , piece_id=IQPTM37ESAZZJELY4RLPBLBMEFPFYXP63SDVK47P7Y3HN3JKIYEQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.476121286 (396h51m12.246762621s) [noise=true , address=77.90.31.10:4547 , piece_id=GX2EF5YML4GXZ2GH5J54YBV7CGE5JKMC63MECWAMDCZYUA6QUE5A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.486869289 (396h51m12.257510624s) [noise=true , address=77.90.31.17:4789 , piece_id=XS44AT7VLC7G6VVVV4HVZHNT7YTYWLAC2EOPDPDSUQ6BNMXQL2EA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.491485340 (396h51m12.262126695s) [noise=true , address=77.90.31.18:4834 , piece_id=7BABB2K6Z5ROMH7GL4ZS3NMMTCMMKGAK3YQRW45BOQ2YSLVZ6JQQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.491802476 (396h51m12.262443811s) [noise=true , address=77.90.31.16:4741 , piece_id=XGSTR3METENXDLWEX5V65WJJPVLYCTF5FNN6OWOLNQ35EMYF4YBQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.492708459 (396h51m12.263349785s) [noise=true , address=77.90.31.21:4912 , piece_id=5ILPD7GX2AGAQXJLQQ5MLKVQOB6I5Q37RQV33EGGSUZIS57WIY2A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.495223731 (396h51m12.265865087s) [noise=true , address=77.90.31.14:4674 , piece_id=6HVO3E5TKEL5JSCA3AKOS3F6NCPD34L3ACAJGCBKDXMJQNBK2Y2Q, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.496454086 (396h51m12.267095441s) [noise=true , address=77.90.31.13:4638 , piece_id=OLHXLVAS74AXSME7T242FXYDRNY3TAASGFWRZG2DQ7Y7TY45M6IQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:02.845874616 (396h51m37.616515951s) [noise=true , address=45.91.95.4:4028 , piece_id=LNOPV7BHCMO3EAEXSYHXMHR4GJ3277WDWMFZMBE6DAP32RV2G26Q, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:08.932651945 (396h51m43.70329328s) [noise=true , address=104.36.82.44:4875 , piece_id=NPZIJRT65NIYURDZOEQVZW5EJPCDGPL4RVKRQJ4EURFQDJXQZJMA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:17.144956582 (396h51m51.915597917s) [noise=true , address=104.36.82.17:4340 , piece_id=7SQQWWMV5Z6JVSOQKHJXCJAECAPNYVSXTNCWH3I2JLF6GEXAUWHQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:33.266596631 (396h52m8.037237966s) [noise=true , address=45.91.95.5:4033 , piece_id=GOBAVQTDDZOWQHJGUVIKCRNPMEC7WC5GW5BYEQVAYDJKQK7KXAIQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:34.385712938 (396h52m9.156354273s) [noise=true , address=104.36.82.36:4628 , piece_id=WW4Z3U3GIMLRET2K2KJ4D255ROXIQH5DZBZOJQMGU62YEXKZ6G3A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385742234 (396h52m9.156383569s) [noise=true , address=104.36.82.38:4709 , piece_id=CB52JNMPPONRIHQXMBCL4NVC23ONSTA2T4ZVYEU67A3VQZXMMLWA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1Ad77dgyCKuYL8o9Ac1WrMfk7pG1mgkakcVXwZa2QHD6KG5sdH, address:104.36.82.38:4709): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385815304 (396h52m9.156456639s) [noise=true , address=45.91.95.3:4018 , piece_id=WEMOWEK3L6A3GLWZBG4BN4JTP6PWPYZ5ATAODXEL6AAJHE2XA7AA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:12qwhpXGKbp4BZ5xTgzYZyw297QcYcrTWo3zUeXVecXsfRqxB6c, address:45.91.95.3:4018): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385853136 (396h52m9.156494471s) [noise=true , address=104.36.82.60:4339 , piece_id=VFM6GGQB3WJ3CWFXPSJN5BL2FQUTLNHURJRFBUT3XAD5CI6H5YYA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385905035 (396h52m9.15654637s) [noise=true , address=45.91.95.2:4008 , piece_id=WGWH2DZO6DGMDZBAANIDLWHR6MOCHHUZQREZAZZGGS4SE3ILP74A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1WTvNJu1X2JaZAT9x58hxgmx849MyJWXWVsbwe8SfQexjdnJAF, address:45.91.95.2:4008): protocol: write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection; write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection Jul 9 22:02:34.385937026 (396h52m9.156578421s) [seg_pos={0 0}, upload=317505]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 22:02:34.637872527 (396h52m9.408513862s) [upload=317505]: Done uploading segments... Jul 9 22:02:34.637897874 (396h52m9.40853921s) [upload=317505]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 20.981127252s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001305481s FAIL (2025-07-09 22:04:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001319487s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.......X...............X..X....X........X.........X.........] duration: 39.853937186s FAIL (2025-07-09 22:03:58 UTC) orbiter warning: context deadline exceeded Duration: 39.854027197s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [X....X...............X..X.X...X...X.......X............X.X..] duration: 48.689311569s FAIL (2025-07-09 22:03:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 17505]: Done starting upload Jul 9 22:01:35.031454414 (396h51m9.802095779s) [upload=317505]: Uploading segments... Jul 9 22:01:35.031760129 (396h51m9.802401504s) [upload=317505]: Got next segment. Inline: false Jul 9 22:01:35.031826716 (396h51m9.802468081s) [upload=317505]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:01:35.031834370 (396h51m9.802475725s) [upload=317505]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:01:35.482978326 (396h51m10.253619662s) [seg_pos={0 0}, upload=317505]: Begin upload segment... Jul 9 22:01:35.483027780 (396h51m10.253669115s) [seg_pos={0 0}, upload=317505]: Done begin upload segment. Jul 9 22:01:35.483053139 (396h51m10.253694474s) [noise=true , address=77.90.31.16:4741 , piece_id=XGSTR3METENXDLWEX5V65WJJPVLYCTF5FNN6OWOLNQ35EMYF4YBQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483088858 (396h51m10.253730193s) [noise=true , address=104.36.82.17:4340 , piece_id=7SQQWWMV5Z6JVSOQKHJXCJAECAPNYVSXTNCWH3I2JLF6GEXAUWHQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483512137 (396h51m10.254153472s) [noise=true , address=79.127.219.38:4013 , piece_id=IQPTM37ESAZZJELY4RLPBLBMEFPFYXP63SDVK47P7Y3HN3JKIYEQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483526023 (396h51m10.254167359s) [noise=true , address=45.91.95.3:4018 , piece_id=WEMOWEK3L6A3GLWZBG4BN4JTP6PWPYZ5ATAODXEL6AAJHE2XA7AA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483777826 (396h51m10.254419161s) [noise=true , address=104.36.82.38:4709 , piece_id=CB52JNMPPONRIHQXMBCL4NVC23ONSTA2T4ZVYEU67A3VQZXMMLWA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483903696 (396h51m10.254545031s) [noise=true , address=77.90.31.17:4789 , piece_id=XS44AT7VLC7G6VVVV4HVZHNT7YTYWLAC2EOPDPDSUQ6BNMXQL2EA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484036110 (396h51m10.254677435s) [noise=true , address=45.91.95.5:4033 , piece_id=GOBAVQTDDZOWQHJGUVIKCRNPMEC7WC5GW5BYEQVAYDJKQK7KXAIQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484318500 (396h51m10.254959835s) [noise=true , address=77.90.31.10:4547 , piece_id=GX2EF5YML4GXZ2GH5J54YBV7CGE5JKMC63MECWAMDCZYUA6QUE5A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484411849 (396h51m10.255053184s) [noise=true , address=77.90.31.14:4674 , piece_id=6HVO3E5TKEL5JSCA3AKOS3F6NCPD34L3ACAJGCBKDXMJQNBK2Y2Q, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484600008 (396h51m10.255241343s) [noise=true , address=77.90.31.13:4638 , piece_id=OLHXLVAS74AXSME7T242FXYDRNY3TAASGFWRZG2DQ7Y7TY45M6IQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484754364 (396h51m10.255395699s) [noise=true , address=77.90.31.18:4834 , piece_id=7BABB2K6Z5ROMH7GL4ZS3NMMTCMMKGAK3YQRW45BOQ2YSLVZ6JQQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484809600 (396h51m10.255450935s) [noise=true , address=45.91.95.2:4008 , piece_id=WGWH2DZO6DGMDZBAANIDLWHR6MOCHHUZQREZAZZGGS4SE3ILP74A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485011466 (396h51m10.255652791s) [noise=true , address=45.91.95.4:4028 , piece_id=LNOPV7BHCMO3EAEXSYHXMHR4GJ3277WDWMFZMBE6DAP32RV2G26Q, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485075679 (396h51m10.255717014s) [noise=true , address=79.127.220.98:4416 , piece_id=JMUXH2XXYATER2XYXWQMNOWSMVYVSXKCYGH5MHRIZ2XU2T4WKQZQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485387596 (396h51m10.256028931s) [noise=true , address=77.90.31.21:4912 , piece_id=5ILPD7GX2AGAQXJLQQ5MLKVQOB6I5Q37RQV33EGGSUZIS57WIY2A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485694533 (396h51m10.256335868s) [noise=true , address=104.36.82.60:4339 , piece_id=VFM6GGQB3WJ3CWFXPSJN5BL2FQUTLNHURJRFBUT3XAD5CI6H5YYA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.486011620 (396h51m10.256652955s) [noise=true , address=104.36.82.36:4628 , piece_id=WW4Z3U3GIMLRET2K2KJ4D255ROXIQH5DZBZOJQMGU62YEXKZ6G3A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.486306593 (396h51m10.256947928s) [noise=true , address=104.36.82.44:4875 , piece_id=NPZIJRT65NIYURDZOEQVZW5EJPCDGPL4RVKRQJ4EURFQDJXQZJMA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:36.891865578 (396h51m11.662506913s) [upload=317505]: Next returned nil segment Jul 9 22:01:36.891886257 (396h51m11.662527582s) [upload=317505]: Waiting for error group managing segments... Jul 9 22:01:37.305485395 (396h51m12.07612673s) [noise=true , address=79.127.220.98:4416 , piece_id=JMUXH2XXYATER2XYXWQMNOWSMVYVSXKCYGH5MHRIZ2XU2T4WKQZQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.473014492 (396h51m12.243655827s) [noise=true , address=79.127.219.38:4013 , piece_id=IQPTM37ESAZZJELY4RLPBLBMEFPFYXP63SDVK47P7Y3HN3JKIYEQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.476121286 (396h51m12.246762621s) [noise=true , address=77.90.31.10:4547 , piece_id=GX2EF5YML4GXZ2GH5J54YBV7CGE5JKMC63MECWAMDCZYUA6QUE5A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.486869289 (396h51m12.257510624s) [noise=true , address=77.90.31.17:4789 , piece_id=XS44AT7VLC7G6VVVV4HVZHNT7YTYWLAC2EOPDPDSUQ6BNMXQL2EA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.491485340 (396h51m12.262126695s) [noise=true , address=77.90.31.18:4834 , piece_id=7BABB2K6Z5ROMH7GL4ZS3NMMTCMMKGAK3YQRW45BOQ2YSLVZ6JQQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.491802476 (396h51m12.262443811s) [noise=true , address=77.90.31.16:4741 , piece_id=XGSTR3METENXDLWEX5V65WJJPVLYCTF5FNN6OWOLNQ35EMYF4YBQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.492708459 (396h51m12.263349785s) [noise=true , address=77.90.31.21:4912 , piece_id=5ILPD7GX2AGAQXJLQQ5MLKVQOB6I5Q37RQV33EGGSUZIS57WIY2A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.495223731 (396h51m12.265865087s) [noise=true , address=77.90.31.14:4674 , piece_id=6HVO3E5TKEL5JSCA3AKOS3F6NCPD34L3ACAJGCBKDXMJQNBK2Y2Q, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.496454086 (396h51m12.267095441s) [noise=true , address=77.90.31.13:4638 , piece_id=OLHXLVAS74AXSME7T242FXYDRNY3TAASGFWRZG2DQ7Y7TY45M6IQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:02.845874616 (396h51m37.616515951s) [noise=true , address=45.91.95.4:4028 , piece_id=LNOPV7BHCMO3EAEXSYHXMHR4GJ3277WDWMFZMBE6DAP32RV2G26Q, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:08.932651945 (396h51m43.70329328s) [noise=true , address=104.36.82.44:4875 , piece_id=NPZIJRT65NIYURDZOEQVZW5EJPCDGPL4RVKRQJ4EURFQDJXQZJMA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:17.144956582 (396h51m51.915597917s) [noise=true , address=104.36.82.17:4340 , piece_id=7SQQWWMV5Z6JVSOQKHJXCJAECAPNYVSXTNCWH3I2JLF6GEXAUWHQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:33.266596631 (396h52m8.037237966s) [noise=true , address=45.91.95.5:4033 , piece_id=GOBAVQTDDZOWQHJGUVIKCRNPMEC7WC5GW5BYEQVAYDJKQK7KXAIQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:34.385712938 (396h52m9.156354273s) [noise=true , address=104.36.82.36:4628 , piece_id=WW4Z3U3GIMLRET2K2KJ4D255ROXIQH5DZBZOJQMGU62YEXKZ6G3A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385742234 (396h52m9.156383569s) [noise=true , address=104.36.82.38:4709 , piece_id=CB52JNMPPONRIHQXMBCL4NVC23ONSTA2T4ZVYEU67A3VQZXMMLWA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1Ad77dgyCKuYL8o9Ac1WrMfk7pG1mgkakcVXwZa2QHD6KG5sdH, address:104.36.82.38:4709): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385815304 (396h52m9.156456639s) [noise=true , address=45.91.95.3:4018 , piece_id=WEMOWEK3L6A3GLWZBG4BN4JTP6PWPYZ5ATAODXEL6AAJHE2XA7AA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:12qwhpXGKbp4BZ5xTgzYZyw297QcYcrTWo3zUeXVecXsfRqxB6c, address:45.91.95.3:4018): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385853136 (396h52m9.156494471s) [noise=true , address=104.36.82.60:4339 , piece_id=VFM6GGQB3WJ3CWFXPSJN5BL2FQUTLNHURJRFBUT3XAD5CI6H5YYA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385905035 (396h52m9.15654637s) [noise=true , address=45.91.95.2:4008 , piece_id=WGWH2DZO6DGMDZBAANIDLWHR6MOCHHUZQREZAZZGGS4SE3ILP74A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1WTvNJu1X2JaZAT9x58hxgmx849MyJWXWVsbwe8SfQexjdnJAF, address:45.91.95.2:4008): protocol: write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection; write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection Jul 9 22:02:34.385937026 (396h52m9.156578421s) [seg_pos={0 0}, upload=317505]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 22:02:34.637872527 (396h52m9.408513862s) [upload=317505]: Done uploading segments... Jul 9 22:02:34.637897874 (396h52m9.40853921s) [upload=317505]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 48.689337588s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 4.891458962s FAIL (2025-07-09 22:03:37 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 4.891488918s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.000671642s FAIL (2025-07-09 22:03:27 UTC) orbiter warning: uplink: encryption: context deadline exceeded Duration: 30.00069193s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [.X....X...............X..X.X...X...X.......X............X.X.] duration: 23.805029763s FAIL (2025-07-09 22:02:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 17505]: Done starting upload Jul 9 22:01:35.031454414 (396h51m9.802095779s) [upload=317505]: Uploading segments... Jul 9 22:01:35.031760129 (396h51m9.802401504s) [upload=317505]: Got next segment. Inline: false Jul 9 22:01:35.031826716 (396h51m9.802468081s) [upload=317505]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:01:35.031834370 (396h51m9.802475725s) [upload=317505]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:01:35.482978326 (396h51m10.253619662s) [seg_pos={0 0}, upload=317505]: Begin upload segment... Jul 9 22:01:35.483027780 (396h51m10.253669115s) [seg_pos={0 0}, upload=317505]: Done begin upload segment. Jul 9 22:01:35.483053139 (396h51m10.253694474s) [noise=true , address=77.90.31.16:4741 , piece_id=XGSTR3METENXDLWEX5V65WJJPVLYCTF5FNN6OWOLNQ35EMYF4YBQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483088858 (396h51m10.253730193s) [noise=true , address=104.36.82.17:4340 , piece_id=7SQQWWMV5Z6JVSOQKHJXCJAECAPNYVSXTNCWH3I2JLF6GEXAUWHQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483512137 (396h51m10.254153472s) [noise=true , address=79.127.219.38:4013 , piece_id=IQPTM37ESAZZJELY4RLPBLBMEFPFYXP63SDVK47P7Y3HN3JKIYEQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483526023 (396h51m10.254167359s) [noise=true , address=45.91.95.3:4018 , piece_id=WEMOWEK3L6A3GLWZBG4BN4JTP6PWPYZ5ATAODXEL6AAJHE2XA7AA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483777826 (396h51m10.254419161s) [noise=true , address=104.36.82.38:4709 , piece_id=CB52JNMPPONRIHQXMBCL4NVC23ONSTA2T4ZVYEU67A3VQZXMMLWA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483903696 (396h51m10.254545031s) [noise=true , address=77.90.31.17:4789 , piece_id=XS44AT7VLC7G6VVVV4HVZHNT7YTYWLAC2EOPDPDSUQ6BNMXQL2EA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484036110 (396h51m10.254677435s) [noise=true , address=45.91.95.5:4033 , piece_id=GOBAVQTDDZOWQHJGUVIKCRNPMEC7WC5GW5BYEQVAYDJKQK7KXAIQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484318500 (396h51m10.254959835s) [noise=true , address=77.90.31.10:4547 , piece_id=GX2EF5YML4GXZ2GH5J54YBV7CGE5JKMC63MECWAMDCZYUA6QUE5A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484411849 (396h51m10.255053184s) [noise=true , address=77.90.31.14:4674 , piece_id=6HVO3E5TKEL5JSCA3AKOS3F6NCPD34L3ACAJGCBKDXMJQNBK2Y2Q, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484600008 (396h51m10.255241343s) [noise=true , address=77.90.31.13:4638 , piece_id=OLHXLVAS74AXSME7T242FXYDRNY3TAASGFWRZG2DQ7Y7TY45M6IQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484754364 (396h51m10.255395699s) [noise=true , address=77.90.31.18:4834 , piece_id=7BABB2K6Z5ROMH7GL4ZS3NMMTCMMKGAK3YQRW45BOQ2YSLVZ6JQQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484809600 (396h51m10.255450935s) [noise=true , address=45.91.95.2:4008 , piece_id=WGWH2DZO6DGMDZBAANIDLWHR6MOCHHUZQREZAZZGGS4SE3ILP74A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485011466 (396h51m10.255652791s) [noise=true , address=45.91.95.4:4028 , piece_id=LNOPV7BHCMO3EAEXSYHXMHR4GJ3277WDWMFZMBE6DAP32RV2G26Q, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485075679 (396h51m10.255717014s) [noise=true , address=79.127.220.98:4416 , piece_id=JMUXH2XXYATER2XYXWQMNOWSMVYVSXKCYGH5MHRIZ2XU2T4WKQZQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485387596 (396h51m10.256028931s) [noise=true , address=77.90.31.21:4912 , piece_id=5ILPD7GX2AGAQXJLQQ5MLKVQOB6I5Q37RQV33EGGSUZIS57WIY2A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485694533 (396h51m10.256335868s) [noise=true , address=104.36.82.60:4339 , piece_id=VFM6GGQB3WJ3CWFXPSJN5BL2FQUTLNHURJRFBUT3XAD5CI6H5YYA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.486011620 (396h51m10.256652955s) [noise=true , address=104.36.82.36:4628 , piece_id=WW4Z3U3GIMLRET2K2KJ4D255ROXIQH5DZBZOJQMGU62YEXKZ6G3A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.486306593 (396h51m10.256947928s) [noise=true , address=104.36.82.44:4875 , piece_id=NPZIJRT65NIYURDZOEQVZW5EJPCDGPL4RVKRQJ4EURFQDJXQZJMA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:36.891865578 (396h51m11.662506913s) [upload=317505]: Next returned nil segment Jul 9 22:01:36.891886257 (396h51m11.662527582s) [upload=317505]: Waiting for error group managing segments... Jul 9 22:01:37.305485395 (396h51m12.07612673s) [noise=true , address=79.127.220.98:4416 , piece_id=JMUXH2XXYATER2XYXWQMNOWSMVYVSXKCYGH5MHRIZ2XU2T4WKQZQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.473014492 (396h51m12.243655827s) [noise=true , address=79.127.219.38:4013 , piece_id=IQPTM37ESAZZJELY4RLPBLBMEFPFYXP63SDVK47P7Y3HN3JKIYEQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.476121286 (396h51m12.246762621s) [noise=true , address=77.90.31.10:4547 , piece_id=GX2EF5YML4GXZ2GH5J54YBV7CGE5JKMC63MECWAMDCZYUA6QUE5A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.486869289 (396h51m12.257510624s) [noise=true , address=77.90.31.17:4789 , piece_id=XS44AT7VLC7G6VVVV4HVZHNT7YTYWLAC2EOPDPDSUQ6BNMXQL2EA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.491485340 (396h51m12.262126695s) [noise=true , address=77.90.31.18:4834 , piece_id=7BABB2K6Z5ROMH7GL4ZS3NMMTCMMKGAK3YQRW45BOQ2YSLVZ6JQQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.491802476 (396h51m12.262443811s) [noise=true , address=77.90.31.16:4741 , piece_id=XGSTR3METENXDLWEX5V65WJJPVLYCTF5FNN6OWOLNQ35EMYF4YBQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.492708459 (396h51m12.263349785s) [noise=true , address=77.90.31.21:4912 , piece_id=5ILPD7GX2AGAQXJLQQ5MLKVQOB6I5Q37RQV33EGGSUZIS57WIY2A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.495223731 (396h51m12.265865087s) [noise=true , address=77.90.31.14:4674 , piece_id=6HVO3E5TKEL5JSCA3AKOS3F6NCPD34L3ACAJGCBKDXMJQNBK2Y2Q, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.496454086 (396h51m12.267095441s) [noise=true , address=77.90.31.13:4638 , piece_id=OLHXLVAS74AXSME7T242FXYDRNY3TAASGFWRZG2DQ7Y7TY45M6IQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:02.845874616 (396h51m37.616515951s) [noise=true , address=45.91.95.4:4028 , piece_id=LNOPV7BHCMO3EAEXSYHXMHR4GJ3277WDWMFZMBE6DAP32RV2G26Q, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:08.932651945 (396h51m43.70329328s) [noise=true , address=104.36.82.44:4875 , piece_id=NPZIJRT65NIYURDZOEQVZW5EJPCDGPL4RVKRQJ4EURFQDJXQZJMA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:17.144956582 (396h51m51.915597917s) [noise=true , address=104.36.82.17:4340 , piece_id=7SQQWWMV5Z6JVSOQKHJXCJAECAPNYVSXTNCWH3I2JLF6GEXAUWHQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:33.266596631 (396h52m8.037237966s) [noise=true , address=45.91.95.5:4033 , piece_id=GOBAVQTDDZOWQHJGUVIKCRNPMEC7WC5GW5BYEQVAYDJKQK7KXAIQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:34.385712938 (396h52m9.156354273s) [noise=true , address=104.36.82.36:4628 , piece_id=WW4Z3U3GIMLRET2K2KJ4D255ROXIQH5DZBZOJQMGU62YEXKZ6G3A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385742234 (396h52m9.156383569s) [noise=true , address=104.36.82.38:4709 , piece_id=CB52JNMPPONRIHQXMBCL4NVC23ONSTA2T4ZVYEU67A3VQZXMMLWA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1Ad77dgyCKuYL8o9Ac1WrMfk7pG1mgkakcVXwZa2QHD6KG5sdH, address:104.36.82.38:4709): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385815304 (396h52m9.156456639s) [noise=true , address=45.91.95.3:4018 , piece_id=WEMOWEK3L6A3GLWZBG4BN4JTP6PWPYZ5ATAODXEL6AAJHE2XA7AA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:12qwhpXGKbp4BZ5xTgzYZyw297QcYcrTWo3zUeXVecXsfRqxB6c, address:45.91.95.3:4018): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385853136 (396h52m9.156494471s) [noise=true , address=104.36.82.60:4339 , piece_id=VFM6GGQB3WJ3CWFXPSJN5BL2FQUTLNHURJRFBUT3XAD5CI6H5YYA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385905035 (396h52m9.15654637s) [noise=true , address=45.91.95.2:4008 , piece_id=WGWH2DZO6DGMDZBAANIDLWHR6MOCHHUZQREZAZZGGS4SE3ILP74A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1WTvNJu1X2JaZAT9x58hxgmx849MyJWXWVsbwe8SfQexjdnJAF, address:45.91.95.2:4008): protocol: write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection; write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection Jul 9 22:02:34.385937026 (396h52m9.156578421s) [seg_pos={0 0}, upload=317505]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 22:02:34.637872527 (396h52m9.408513862s) [upload=317505]: Done uploading segments... Jul 9 22:02:34.637897874 (396h52m9.40853921s) [upload=317505]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 23.805055983s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [........X...............X..X....X........X.........X........] duration: 35.047703939s FAIL (2025-07-09 22:02:56 UTC) orbiter warning: context deadline exceeded Duration: 35.047860819s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [..X....X...............X..X.X...X...X.......X............X.X] duration: 1m0.254013972s FAIL (2025-07-09 22:02:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 17505]: Done starting upload Jul 9 22:01:35.031454414 (396h51m9.802095779s) [upload=317505]: Uploading segments... Jul 9 22:01:35.031760129 (396h51m9.802401504s) [upload=317505]: Got next segment. Inline: false Jul 9 22:01:35.031826716 (396h51m9.802468081s) [upload=317505]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 9 22:01:35.031834370 (396h51m9.802475725s) [upload=317505]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 9 22:01:35.482978326 (396h51m10.253619662s) [seg_pos={0 0}, upload=317505]: Begin upload segment... Jul 9 22:01:35.483027780 (396h51m10.253669115s) [seg_pos={0 0}, upload=317505]: Done begin upload segment. Jul 9 22:01:35.483053139 (396h51m10.253694474s) [noise=true , address=77.90.31.16:4741 , piece_id=XGSTR3METENXDLWEX5V65WJJPVLYCTF5FNN6OWOLNQ35EMYF4YBQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483088858 (396h51m10.253730193s) [noise=true , address=104.36.82.17:4340 , piece_id=7SQQWWMV5Z6JVSOQKHJXCJAECAPNYVSXTNCWH3I2JLF6GEXAUWHQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483512137 (396h51m10.254153472s) [noise=true , address=79.127.219.38:4013 , piece_id=IQPTM37ESAZZJELY4RLPBLBMEFPFYXP63SDVK47P7Y3HN3JKIYEQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483526023 (396h51m10.254167359s) [noise=true , address=45.91.95.3:4018 , piece_id=WEMOWEK3L6A3GLWZBG4BN4JTP6PWPYZ5ATAODXEL6AAJHE2XA7AA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483777826 (396h51m10.254419161s) [noise=true , address=104.36.82.38:4709 , piece_id=CB52JNMPPONRIHQXMBCL4NVC23ONSTA2T4ZVYEU67A3VQZXMMLWA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.483903696 (396h51m10.254545031s) [noise=true , address=77.90.31.17:4789 , piece_id=XS44AT7VLC7G6VVVV4HVZHNT7YTYWLAC2EOPDPDSUQ6BNMXQL2EA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484036110 (396h51m10.254677435s) [noise=true , address=45.91.95.5:4033 , piece_id=GOBAVQTDDZOWQHJGUVIKCRNPMEC7WC5GW5BYEQVAYDJKQK7KXAIQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484318500 (396h51m10.254959835s) [noise=true , address=77.90.31.10:4547 , piece_id=GX2EF5YML4GXZ2GH5J54YBV7CGE5JKMC63MECWAMDCZYUA6QUE5A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484411849 (396h51m10.255053184s) [noise=true , address=77.90.31.14:4674 , piece_id=6HVO3E5TKEL5JSCA3AKOS3F6NCPD34L3ACAJGCBKDXMJQNBK2Y2Q, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484600008 (396h51m10.255241343s) [noise=true , address=77.90.31.13:4638 , piece_id=OLHXLVAS74AXSME7T242FXYDRNY3TAASGFWRZG2DQ7Y7TY45M6IQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484754364 (396h51m10.255395699s) [noise=true , address=77.90.31.18:4834 , piece_id=7BABB2K6Z5ROMH7GL4ZS3NMMTCMMKGAK3YQRW45BOQ2YSLVZ6JQQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.484809600 (396h51m10.255450935s) [noise=true , address=45.91.95.2:4008 , piece_id=WGWH2DZO6DGMDZBAANIDLWHR6MOCHHUZQREZAZZGGS4SE3ILP74A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485011466 (396h51m10.255652791s) [noise=true , address=45.91.95.4:4028 , piece_id=LNOPV7BHCMO3EAEXSYHXMHR4GJ3277WDWMFZMBE6DAP32RV2G26Q, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485075679 (396h51m10.255717014s) [noise=true , address=79.127.220.98:4416 , piece_id=JMUXH2XXYATER2XYXWQMNOWSMVYVSXKCYGH5MHRIZ2XU2T4WKQZQ, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485387596 (396h51m10.256028931s) [noise=true , address=77.90.31.21:4912 , piece_id=5ILPD7GX2AGAQXJLQQ5MLKVQOB6I5Q37RQV33EGGSUZIS57WIY2A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.485694533 (396h51m10.256335868s) [noise=true , address=104.36.82.60:4339 , piece_id=VFM6GGQB3WJ3CWFXPSJN5BL2FQUTLNHURJRFBUT3XAD5CI6H5YYA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.486011620 (396h51m10.256652955s) [noise=true , address=104.36.82.36:4628 , piece_id=WW4Z3U3GIMLRET2K2KJ4D255ROXIQH5DZBZOJQMGU62YEXKZ6G3A, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:35.486306593 (396h51m10.256947928s) [noise=true , address=104.36.82.44:4875 , piece_id=NPZIJRT65NIYURDZOEQVZW5EJPCDGPL4RVKRQJ4EURFQDJXQZJMA, seg_pos={0 0}, upload=317505]: Uploading piece... Jul 9 22:01:36.891865578 (396h51m11.662506913s) [upload=317505]: Next returned nil segment Jul 9 22:01:36.891886257 (396h51m11.662527582s) [upload=317505]: Waiting for error group managing segments... Jul 9 22:01:37.305485395 (396h51m12.07612673s) [noise=true , address=79.127.220.98:4416 , piece_id=JMUXH2XXYATER2XYXWQMNOWSMVYVSXKCYGH5MHRIZ2XU2T4WKQZQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.473014492 (396h51m12.243655827s) [noise=true , address=79.127.219.38:4013 , piece_id=IQPTM37ESAZZJELY4RLPBLBMEFPFYXP63SDVK47P7Y3HN3JKIYEQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.476121286 (396h51m12.246762621s) [noise=true , address=77.90.31.10:4547 , piece_id=GX2EF5YML4GXZ2GH5J54YBV7CGE5JKMC63MECWAMDCZYUA6QUE5A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.486869289 (396h51m12.257510624s) [noise=true , address=77.90.31.17:4789 , piece_id=XS44AT7VLC7G6VVVV4HVZHNT7YTYWLAC2EOPDPDSUQ6BNMXQL2EA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.491485340 (396h51m12.262126695s) [noise=true , address=77.90.31.18:4834 , piece_id=7BABB2K6Z5ROMH7GL4ZS3NMMTCMMKGAK3YQRW45BOQ2YSLVZ6JQQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.491802476 (396h51m12.262443811s) [noise=true , address=77.90.31.16:4741 , piece_id=XGSTR3METENXDLWEX5V65WJJPVLYCTF5FNN6OWOLNQ35EMYF4YBQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.492708459 (396h51m12.263349785s) [noise=true , address=77.90.31.21:4912 , piece_id=5ILPD7GX2AGAQXJLQQ5MLKVQOB6I5Q37RQV33EGGSUZIS57WIY2A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.495223731 (396h51m12.265865087s) [noise=true , address=77.90.31.14:4674 , piece_id=6HVO3E5TKEL5JSCA3AKOS3F6NCPD34L3ACAJGCBKDXMJQNBK2Y2Q, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:01:37.496454086 (396h51m12.267095441s) [noise=true , address=77.90.31.13:4638 , piece_id=OLHXLVAS74AXSME7T242FXYDRNY3TAASGFWRZG2DQ7Y7TY45M6IQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:02.845874616 (396h51m37.616515951s) [noise=true , address=45.91.95.4:4028 , piece_id=LNOPV7BHCMO3EAEXSYHXMHR4GJ3277WDWMFZMBE6DAP32RV2G26Q, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:08.932651945 (396h51m43.70329328s) [noise=true , address=104.36.82.44:4875 , piece_id=NPZIJRT65NIYURDZOEQVZW5EJPCDGPL4RVKRQJ4EURFQDJXQZJMA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:17.144956582 (396h51m51.915597917s) [noise=true , address=104.36.82.17:4340 , piece_id=7SQQWWMV5Z6JVSOQKHJXCJAECAPNYVSXTNCWH3I2JLF6GEXAUWHQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:33.266596631 (396h52m8.037237966s) [noise=true , address=45.91.95.5:4033 , piece_id=GOBAVQTDDZOWQHJGUVIKCRNPMEC7WC5GW5BYEQVAYDJKQK7KXAIQ, seg_pos={0 0}, upload=317505]: Done uploading piece. err: <nil> Jul 9 22:02:34.385712938 (396h52m9.156354273s) [noise=true , address=104.36.82.36:4628 , piece_id=WW4Z3U3GIMLRET2K2KJ4D255ROXIQH5DZBZOJQMGU62YEXKZ6G3A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1y9h3t9CQKukCffvjBC6XugNBCHguXyANBSqj6Qr98sqmPiqsp, address:104.36.82.36:4628): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385742234 (396h52m9.156383569s) [noise=true , address=104.36.82.38:4709 , piece_id=CB52JNMPPONRIHQXMBCL4NVC23ONSTA2T4ZVYEU67A3VQZXMMLWA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1Ad77dgyCKuYL8o9Ac1WrMfk7pG1mgkakcVXwZa2QHD6KG5sdH, address:104.36.82.38:4709): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385815304 (396h52m9.156456639s) [noise=true , address=45.91.95.3:4018 , piece_id=WEMOWEK3L6A3GLWZBG4BN4JTP6PWPYZ5ATAODXEL6AAJHE2XA7AA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:12qwhpXGKbp4BZ5xTgzYZyw297QcYcrTWo3zUeXVecXsfRqxB6c, address:45.91.95.3:4018): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385853136 (396h52m9.156494471s) [noise=true , address=104.36.82.60:4339 , piece_id=VFM6GGQB3WJ3CWFXPSJN5BL2FQUTLNHURJRFBUT3XAD5CI6H5YYA, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R, address:104.36.82.60:4339): protocol: expected piece hash; context deadline exceeded; EOF Jul 9 22:02:34.385905035 (396h52m9.15654637s) [noise=true , address=45.91.95.2:4008 , piece_id=WGWH2DZO6DGMDZBAANIDLWHR6MOCHHUZQREZAZZGGS4SE3ILP74A, seg_pos={0 0}, upload=317505]: Done uploading piece. err: ecclient: upload failed (node:1WTvNJu1X2JaZAT9x58hxgmx849MyJWXWVsbwe8SfQexjdnJAF, address:45.91.95.2:4008): protocol: write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection; write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52372->45.91.95.2:4008: use of closed network connection Jul 9 22:02:34.385937026 (396h52m9.156578421s) [seg_pos={0 0}, upload=317505]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 9 22:02:34.637872527 (396h52m9.408513862s) [upload=317505]: Done uploading segments... Jul 9 22:02:34.637897874 (396h52m9.40853921s) [upload=317505]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.25403967s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.........X...............X..X....X........X.........X.......] duration: 31.352242993s FAIL (2025-07-09 22:02:14 UTC) orbiter warning: context deadline exceeded Duration: 31.352261248s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [...X....X...............X..X.X...X...X.......X............X.] duration: 28.466150367s FAIL (2025-07-09 22:01:24 UTC) orbiter warning: ======ERROR====== uplink: context deadline exceeded ======LOGS====== sh batch item: pb.BatchRequestItem_SegmentBegin Jul 9 21:59:57.365491004 (396h49m32.136132339s) [seg_pos={0 0}, upload=317485]: Begin upload segment... Jul 9 21:59:57.365538214 (396h49m32.136179549s) [seg_pos={0 0}, upload=317485]: Done begin upload segment. Jul 9 21:59:57.365554856 (396h49m32.136196191s) [noise=true , address=45.91.95.4:4001 , piece_id=CQ6ELD424VMDVRVC6J3CVSGWCGZD3WPYTKVLANXASZ25OYBD2PDQ, seg_pos={0 0}, upload=317485]: Uploading piece... Jul 9 21:59:57.365563883 (396h49m32.136205218s) [noise=true , address=77.90.31.21:4887 , piece_id=EOI5DVL4X2GBSUKXSYUECS3QNFDA2NMLWLHLSTGOBCVHJIPIALDQ, seg_pos={0 0}, upload=317485]: Uploading piece... Jul 9 21:59:57.365620953 (396h49m32.136262288s) [noise=true , address=104.36.82.38:4710 , piece_id=F2THYGFPVUQJMDLOBZQHVDSMZJN5CPVEGNMBI5VZ2W4KLYWLOJCQ, seg_pos={0 0}, upload=317485]: Uploading piece... Jul 9 21:59:57.365982464 (396h49m32.136623799s) [noise=true , address=77.90.31.14:4677 , piece_id=B34QFJFGT6P3F3FLTCUCGRNN77QYK3CDZ63MG6NL2NTSVGFM4ZXA, seg_pos={0 0}, upload=317485]: Uploading piece... Jul 9 21:59:57.366266678 (396h49m32.136908013s) [noise=true , address=104.36.82.40:4969 , piece_id=E4TFISU72AEYCLQOW44S3VAX6PNYNI6UFNVVEB2IZ2NNCDBZT27Q, seg_pos={0 0}, upload=317485]: Uploading piece... Jul 9 21:59:57.366371769 (396h49m32.137013104s) [noise=true , address=45.91.95.2:4035 , piece_id=IJZZVC5X6CLDTMMQBJEYG4KFFSWWESCEDC36NRLHDHTDT5XU6ZCA, seg_pos={0 0}, upload=317485]: Uploading piece... Jul 9 21:59:57.366643349 (396h49m32.137284674s) [noise=true , address=77.90.31.19:4858 , piece_id=DW2QOKG3LEGUDVMSPMN2DXLYZGUAFU2JRH5R7MLWL4OSLHKBAMXA, seg_pos={0 0}, upload=317485]: Uploading piece... Jul 9 21:59:57.366911571 (396h49m32.137552906s) [noise=true , address=79.127.220.97:4368 , piece_id=5BXO5TDDG72YTM6S3SHXF5MIBYZ6MRMOXFS337NVEKKAGUYVLE6A, seg_pos={0 0}, upload=317485]: Uploading piece... Jul 9 21:59:57.367116163 (396h49m32.137757488s) [noise=true , address=77.90.31.14:4682 , piece_id=7BZTN2CTBAAIRSAX4WGZ3YHJ75NCVS4HZXAGEGTSRGJQQJDJ26TA, seg_pos={0 0}, upload=317485]: Uploading piece... Jul 9 21:59:57.367329601 (396h49m32.137970926s) [noise=true , address=104.36.82.18:4396 , piece_id=NOF4ILCK6CSTCVRNN3EYNWFNVYX2KU5ESFCADU6P3GSXV7ZPPXCQ, seg_pos={0 0}, upload=317485]: Uploading piece... Jul 9 21:59:57.367566244 (396h49m32.138207579s) [noise=true , address=77.90.31.10:4544 , piece_id=TRYZDQ4XRBKX5TTJTFHASIXVMWUSXFD7ASZ3CSBIUQIEICF6WMWQ, seg_pos={0 0}, upload=317485]: Uploading piece... Jul 9 21:59:57.367639785 (396h49m32.13828112s) [noise=true , address=77.90.31.16:4736 , piece_id=LR6X3G74LHHQJCARXWM2DQUME57BBDDHU377PLNXXVE43R3RLUUQ, seg_pos={0 0}, upload=317485]: Uploading piece... Jul 9 21:59:57.367886267 (396h49m32.138527602s) [noise=true , address=104.36.82.36:4651 , piece_id=ULRXB4QI3YQM66SYSI7VJ2ZZJX5R5UK7LPUVDQCZK6YKL32QI57A, seg_pos={0 0}, upload=317485]: Uploading piece... Jul 9 21:59:57.368109754 (396h49m32.138751089s) [noise=true , address=79.127.220.101:4041 , piece_id=PTCRPQZPNCCFH4XSMVGMPFZQAWUZBJG7JIYPFQUAVE6F52D6RPEA, seg_pos={0 0}, upload=317485]: Uploading piece... Jul 9 21:59:57.368339253 (396h49m32.138980588s) [noise=true , address=77.90.31.13:4652 , piece_id=EEUPGMZ5MMQOXKJJ74QQ4B5ILZUZV2TADEK2PPWIVCIP7L35YH4A, seg_pos={0 0}, upload=317485]: Uploading piece... Jul 9 21:59:57.368472368 (396h49m32.139113703s) [noise=true , address=45.91.95.3:4022 , piece_id=RGE5B3IDAZAAQ5BJ3INCJBJVVL2NJN737ADZZNAU5A3BKME7RWHQ, seg_pos={0 0}, upload=317485]: Uploading piece... Jul 9 21:59:57.368690305 (396h49m32.13933164s) [noise=true , address=79.127.219.38:4023 , piece_id=5JZEDEGXA6RHRL7OOMGZU6ICTOXDD6NF2XVRZK6XWQ7VKK6LWYOA, seg_pos={0 0}, upload=317485]: Uploading piece... Jul 9 21:59:57.368909765 (396h49m32.1395511s) [noise=true , address=45.91.95.5:4006 , piece_id=SEIVP7IDEJW6VPN5A7O2VKNCPNNF7NTGJTMDBI2NBBFN5ORNTKSA, seg_pos={0 0}, upload=317485]: Uploading piece... Jul 9 21:59:58.313353166 (396h49m33.083994491s) [noise=true , address=45.91.95.4:4001 , piece_id=CQ6ELD424VMDVRVC6J3CVSGWCGZD3WPYTKVLANXASZ25OYBD2PDQ, seg_pos={0 0}, upload=317485]: Done uploading piece. err: ecclient: upload failed (node:12PdimTdB4ysD3SQ8hKMAGAXkzzC71irLv6rW9FArf5qmrvZ5MY, address:45.91.95.4:4001): protocol: storage node overloaded, request limit: 2000 Jul 9 21:59:58.756065621 (396h49m33.526706956s) [upload=317485]: Next returned nil segment Jul 9 21:59:58.756084979 (396h49m33.526726314s) [upload=317485]: Waiting for error group managing segments... Jul 9 21:59:59.268055194 (396h49m34.038696529s) [noise=true , address=79.127.220.101:4041 , piece_id=PTCRPQZPNCCFH4XSMVGMPFZQAWUZBJG7JIYPFQUAVE6F52D6RPEA, seg_pos={0 0}, upload=317485]: Done uploading piece. err: <nil> Jul 9 21:59:59.345248446 (396h49m34.115889781s) [noise=true , address=79.127.220.97:4368 , piece_id=5BXO5TDDG72YTM6S3SHXF5MIBYZ6MRMOXFS337NVEKKAGUYVLE6A, seg_pos={0 0}, upload=317485]: Done uploading piece. err: <nil> Jul 9 21:59:59.371176971 (396h49m34.141818306s) [noise=true , address=77.90.31.19:4858 , piece_id=DW2QOKG3LEGUDVMSPMN2DXLYZGUAFU2JRH5R7MLWL4OSLHKBAMXA, seg_pos={0 0}, upload=317485]: Done uploading piece. err: <nil> Jul 9 21:59:59.373114388 (396h49m34.143755723s) [noise=true , address=77.90.31.14:4677 , piece_id=B34QFJFGT6P3F3FLTCUCGRNN77QYK3CDZ63MG6NL2NTSVGFM4ZXA, seg_pos={0 0}, upload=317485]: Done uploading piece. err: <nil> Jul 9 21:59:59.373343305 (396h49m34.14398464s) [noise=true , address=77.90.31.16:4736 , piece_id=LR6X3G74LHHQJCARXWM2DQUME57BBDDHU377PLNXXVE43R3RLUUQ, seg_pos={0 0}, upload=317485]: Done uploading piece. err: <nil> Jul 9 21:59:59.373440552 (396h49m34.144081887s) [noise=true , address=77.90.31.14:4682 , piece_id=7BZTN2CTBAAIRSAX4WGZ3YHJ75NCVS4HZXAGEGTSRGJQQJDJ26TA, seg_pos={0 0}, upload=317485]: Done uploading piece. err: <nil> Jul 9 21:59:59.389060125 (396h49m34.15970146s) [noise=true , address=77.90.31.13:4652 , piece_id=EEUPGMZ5MMQOXKJJ74QQ4B5ILZUZV2TADEK2PPWIVCIP7L35YH4A, seg_pos={0 0}, upload=317485]: Done uploading piece. err: <nil> Jul 9 21:59:59.391450517 (396h49m34.162091852s) [noise=true , address=77.90.31.10:4544 , piece_id=TRYZDQ4XRBKX5TTJTFHASIXVMWUSXFD7ASZ3CSBIUQIEICF6WMWQ, seg_pos={0 0}, upload=317485]: Done uploading piece. err: <nil> Jul 9 21:59:59.397080999 (396h49m34.167722364s) [noise=true , address=77.90.31.21:4887 , piece_id=EOI5DVL4X2GBSUKXSYUECS3QNFDA2NMLWLHLSTGOBCVHJIPIALDQ, seg_pos={0 0}, upload=317485]: Done uploading piece. err: <nil> Jul 9 22:00:05.441156519 (396h49m40.211797864s) [noise=true , address=79.127.219.38:4023 , piece_id=5JZEDEGXA6RHRL7OOMGZU6ICTOXDD6NF2XVRZK6XWQ7VKK6LWYOA, seg_pos={0 0}, upload=317485]: Done uploading piece. err: <nil> Jul 9 22:00:19.865539301 (396h49m54.636180637s) [noise=true , address=104.36.82.36:4651 , piece_id=ULRXB4QI3YQM66SYSI7VJ2ZZJX5R5UK7LPUVDQCZK6YKL32QI57A, seg_pos={0 0}, upload=317485]: Done uploading piece. err: <nil> Jul 9 22:00:20.403167714 (396h49m55.173809049s) [noise=true , address=104.36.82.40:4969 , piece_id=E4TFISU72AEYCLQOW44S3VAX6PNYNI6UFNVVEB2IZ2NNCDBZT27Q, seg_pos={0 0}, upload=317485]: Done uploading piece. err: <nil> Jul 9 22:00:22.586466101 (396h49m57.357107436s) [noise=true , address=45.91.95.5:4006 , piece_id=SEIVP7IDEJW6VPN5A7O2VKNCPNNF7NTGJTMDBI2NBBFN5ORNTKSA, seg_pos={0 0}, upload=317485]: Done uploading piece. err: <nil> Jul 9 22:00:54.843272588 (396h50m29.613913924s) [noise=true , address=104.36.82.38:4710 , piece_id=F2THYGFPVUQJMDLOBZQHVDSMZJN5CPVEGNMBI5VZ2W4KLYWLOJCQ, seg_pos={0 0}, upload=317485]: Done uploading piece. err: <nil> Jul 9 22:00:54.843345257 (396h50m29.613986602s) [seg_pos={0 0}, upload=317485]: Segment reached optimal threshold of 14 pieces. Jul 9 22:00:54.843532926 (396h50m29.614174261s) [noise=true , address=45.91.95.2:4035 , piece_id=IJZZVC5X6CLDTMMQBJEYG4KFFSWWESCEDC36NRLHDHTDT5XU6ZCA, seg_pos={0 0}, upload=317485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121Xa3rvgKQhHc5NBufs57Ybnw3fYKUjgRyK32TTwWoi4E8m57o): protocol: write tcp 188.166.225.170:36034->45.91.95.2:4035: use of closed network connection; write tcp 188.166.225.170:36034->45.91.95.2:4035: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36034->45.91.95.2:4035: use of closed network connection Jul 9 22:00:54.843550400 (396h50m29.614191725s) [noise=true , address=104.36.82.18:4396 , piece_id=NOF4ILCK6CSTCVRNN3EYNWFNVYX2KU5ESFCADU6P3GSXV7ZPPXCQ, seg_pos={0 0}, upload=317485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1k5gzJBGWhpARcfiynburgfTqJ2t7sVNs1mAkUfToMG5saxPqx): protocol: expected piece hash; context canceled; EOF Jul 9 22:00:54.843569696 (396h50m29.614211051s) [noise=true , address=45.91.95.3:4022 , piece_id=RGE5B3IDAZAAQ5BJ3INCJBJVVL2NJN737ADZZNAU5A3BKME7RWHQ, seg_pos={0 0}, upload=317485]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16NrDYdTZZYsjkpZhTHuv8ug9bnioi4sg5KNsfM3ocrre8kA1B): protocol: write tcp 188.166.225.170:51474->45.91.95.3:4022: use of closed network connection; write tcp 188.166.225.170:51474->45.91.95.3:4022: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51474->45.91.95.3:4022: use of closed network connection Jul 9 22:00:54.843584315 (396h50m29.61422565s) [seg_pos={0 0}, upload=317485]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Jul 9 22:00:54.843661002 (396h50m29.614302337s) [upload=317485]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 9 22:00:54.843684806 (396h50m29.614326141s) [upload=317485]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 9 22:00:55.149989764 (396h50m29.920631099s) [upload=317485]: Done uploading segments... Jul 9 22:00:55.150006736 (396h50m29.920648072s) [upload=317485]: Upload finished. err: <nil> Duration:* 28.466170486s core uplink us-select-1 us1-placement-20

1
...
8288
8289
8290
8291
8292
...
8609