Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 868049 reports (Page 11632 of 17361)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.640420471s trace: https://jaeger.dev.storj.io/trace/3a1fb019f7c2965d FAIL (2025-12-09 07:21:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 06:55:45.259556946 (87h26m45.130639981s) [upload=76331]: Starting upload Dec 9 06:55:45.259623222 (87h26m45.130706257s) [upload=76331]: Done starting upload Dec 9 06:55:45.259658870 (87h26m45.130741905s) [upload=76331]: Uploading segments... Dec 9 06:55:45.259795500 (87h26m45.130878535s) [upload=76331]: Got next segment. Inline: false Dec 9 06:55:45.259821169 (87h26m45.130904204s) [upload=76331]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 06:55:45.259825137 (87h26m45.130908172s) [upload=76331]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 06:55:45.506331314 (87h26m45.377414359s) [seg_pos={0 0}, upload=76331]: Begin upload segment... Dec 9 06:55:45.506397410 (87h26m45.377480445s) [seg_pos={0 0}, upload=76331]: Done begin upload segment. Dec 9 06:55:45.506452925 (87h26m45.37753596s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.506473765 (87h26m45.3775568s) [noise=true , address=77.90.31.15:4698 , piece_id=2Y3GG5Y2QUJ6PFAYODA4V63N4CQXCTLRNH7LISWCXQWOAVCBSULQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507055783 (87h26m45.378138818s) [noise=true , address=145.79.227.41:4086 , piece_id=GXH4GU22OG3BZQSHTK25AHWPFCYKCQEN72NXEPYXO4ZUIW2RSDEA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507493356 (87h26m45.378576391s) [noise=true , address=145.79.227.29:4942 , piece_id=SMZGQG7OUYGCHPPJLIS3PH5Y3VU65I46D4NYL2PXCBRSPQRSPZEQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507543853 (87h26m45.378626898s) [noise=true , address=145.79.227.16:4909 , piece_id=7TX6RV44XTP4C5DPVEPTSV6ECNGUSXU6G33T6L3AUCFNQLISNHOQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507799740 (87h26m45.378882775s) [noise=true , address=145.79.227.29:4941 , piece_id=UDD375EXXARYRWOXMWZF25HB7VW4YVWQEFESPW4JNNB4MG7QPPSA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507920349 (87h26m45.379003384s) [noise=true , address=145.79.227.31:4797 , piece_id=AW3Q7GXNWAU5T5Z562KBKXYZPJEL5IRR6XB7EZZPZXUJMBQNOSBA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508339767 (87h26m45.379422802s) [noise=true , address=145.79.227.34:4905 , piece_id=RUHC4SR4EO2IF2KEDJNYZOY33EAKF4JR4FTM62NZJCUD2XPO7N6Q, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508566930 (87h26m45.379649965s) [noise=true , address=145.79.227.25:4618 , piece_id=K2Y25ECEZ6SEYOV5NPG2EEWFVN5IFFVM54QPK4ZGWZQRNE3I3BCA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508739269 (87h26m45.379822314s) [noise=true , address=145.79.227.24:4581 , piece_id=WNL2FDFC4N4U5TX522OSVZDEBNO6VVZTL4TOS2C65KHY2CAMHQHA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509134110 (87h26m45.380217145s) [noise=true , address=145.79.227.24:4582 , piece_id=EAMAU5U3OBF5RA762ZYOARJZ4J2J4MXNEKWKY3JWGT3BX76OB6XQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509258507 (87h26m45.380341543s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509546506 (87h26m45.380629541s) [noise=true , address=145.79.227.21:4473 , piece_id=E4G6ZYPNOHCKQ6FYK4I3Y5V36C7RX5TOKVCQKN2VKHFWV5J62VEA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509843851 (87h26m45.380926896s) [noise=true , address=145.79.227.23:4547 , piece_id=UDL6VDEHGJ6Z4HMRU6HM6XIDUCMSWCEECCUH45FT3FQVLUDOEZGQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.510141709 (87h26m45.381224754s) [noise=true , address=145.79.227.38:4977 , piece_id=JZVULFGKR65GE24SFP5U7LHINOZIZ2BXIGQSBTZUSPXEWXEQA32A, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.510428084 (87h26m45.381511129s) [noise=true , address=145.79.227.32:4834 , piece_id=N6VJKJMTE5W3HWGCXDGPTLB3BHSFLWBMESU57UOSWPNBM55DZ4MQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:46.567032201 (87h26m46.438115236s) [noise=true , address=77.90.31.15:4698 , piece_id=2Y3GG5Y2QUJ6PFAYODA4V63N4CQXCTLRNH7LISWCXQWOAVCBSULQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 06:55:47.200741008 (87h26m47.071824053s) [upload=76331]: Next returned nil segment Dec 9 06:55:47.200756667 (87h26m47.071839702s) [upload=76331]: Waiting for error group managing segments... Dec 9 06:55:47.488289311 (87h26m47.359372346s) [noise=true , address=145.79.227.41:4086 , piece_id=GXH4GU22OG3BZQSHTK25AHWPFCYKCQEN72NXEPYXO4ZUIW2RSDEA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.488813710 (87h26m47.359896755s) [noise=true , address=145.79.227.21:4473 , piece_id=E4G6ZYPNOHCKQ6FYK4I3Y5V36C7RX5TOKVCQKN2VKHFWV5J62VEA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.488891818 (87h26m47.359974853s) [noise=true , address=145.79.227.38:4977 , piece_id=JZVULFGKR65GE24SFP5U7LHINOZIZ2BXIGQSBTZUSPXEWXEQA32A, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.489842157 (87h26m47.360925202s) [noise=true , address=145.79.227.24:4582 , piece_id=EAMAU5U3OBF5RA762ZYOARJZ4J2J4MXNEKWKY3JWGT3BX76OB6XQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.489996712 (87h26m47.361079747s) [noise=true , address=145.79.227.32:4834 , piece_id=N6VJKJMTE5W3HWGCXDGPTLB3BHSFLWBMESU57UOSWPNBM55DZ4MQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.495771257 (87h26m47.366854302s) [noise=true , address=145.79.227.34:4905 , piece_id=RUHC4SR4EO2IF2KEDJNYZOY33EAKF4JR4FTM62NZJCUD2XPO7N6Q, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.496288763 (87h26m47.367371808s) [noise=true , address=145.79.227.25:4618 , piece_id=K2Y25ECEZ6SEYOV5NPG2EEWFVN5IFFVM54QPK4ZGWZQRNE3I3BCA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.500744707 (87h26m47.371827752s) [noise=true , address=145.79.227.31:4797 , piece_id=AW3Q7GXNWAU5T5Z562KBKXYZPJEL5IRR6XB7EZZPZXUJMBQNOSBA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.501416546 (87h26m47.372499591s) [noise=true , address=145.79.227.16:4909 , piece_id=7TX6RV44XTP4C5DPVEPTSV6ECNGUSXU6G33T6L3AUCFNQLISNHOQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.501422417 (87h26m47.372505462s) [noise=true , address=145.79.227.23:4547 , piece_id=UDL6VDEHGJ6Z4HMRU6HM6XIDUCMSWCEECCUH45FT3FQVLUDOEZGQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.514960194 (87h26m47.386043249s) [noise=true , address=145.79.227.24:4581 , piece_id=WNL2FDFC4N4U5TX522OSVZDEBNO6VVZTL4TOS2C65KHY2CAMHQHA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516209051 (87h26m47.387292086s) [noise=true , address=145.79.227.29:4941 , piece_id=UDD375EXXARYRWOXMWZF25HB7VW4YVWQEFESPW4JNNB4MG7QPPSA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516598713 (87h26m47.387681758s) [noise=true , address=145.79.227.29:4942 , piece_id=SMZGQG7OUYGCHPPJLIS3PH5Y3VU65I46D4NYL2PXCBRSPQRSPZEQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516607911 (87h26m47.387690946s) [seg_pos={0 0}, upload=76331]: Segment reached optimal threshold of 13 pieces. Dec 9 06:55:47.516785880 (87h26m47.387868915s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u): protocol: expected piece hash; context canceled; EOF Dec 9 06:55:47.516824383 (87h26m47.387907448s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Long tail cancellation due to optimal threshold Dec 9 06:55:47.517129214 (87h26m47.388212259s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 06:55:47.517138261 (87h26m47.388221306s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Long tail cancellation due to optimal threshold Dec 9 06:55:47.517375302 (87h26m47.388458337s) [seg_pos={0 0}, upload=76331]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 06:55:47.517466135 (87h26m47.38854917s) [upload=76331]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 06:55:47.517473488 (87h26m47.388556534s) [upload=76331]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:00:17.830318594 (87h31m17.701401639s) [upload=76331]: Done uploading segments... Dec 9 07:00:17.830402994 (87h31m17.70148605s) [upload=76331]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.640450037s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........X....X............................................] duration: 3.470164167s trace: https://jaeger.dev.storj.io/trace/2c78734d9bae1209 FAIL (2025-12-09 07:21:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.470284858s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [X................X...X...X............................X.....] duration: 5.830878908s trace: https://jaeger.dev.storj.io/trace/2225ff488c04e76d FAIL (2025-12-09 07:21:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.83090603s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.724603265s trace: https://jaeger.dev.storj.io/trace/3a1fb019f7c2965d FAIL (2025-12-09 07:20:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 06:55:45.259556946 (87h26m45.130639981s) [upload=76331]: Starting upload Dec 9 06:55:45.259623222 (87h26m45.130706257s) [upload=76331]: Done starting upload Dec 9 06:55:45.259658870 (87h26m45.130741905s) [upload=76331]: Uploading segments... Dec 9 06:55:45.259795500 (87h26m45.130878535s) [upload=76331]: Got next segment. Inline: false Dec 9 06:55:45.259821169 (87h26m45.130904204s) [upload=76331]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 06:55:45.259825137 (87h26m45.130908172s) [upload=76331]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 06:55:45.506331314 (87h26m45.377414359s) [seg_pos={0 0}, upload=76331]: Begin upload segment... Dec 9 06:55:45.506397410 (87h26m45.377480445s) [seg_pos={0 0}, upload=76331]: Done begin upload segment. Dec 9 06:55:45.506452925 (87h26m45.37753596s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.506473765 (87h26m45.3775568s) [noise=true , address=77.90.31.15:4698 , piece_id=2Y3GG5Y2QUJ6PFAYODA4V63N4CQXCTLRNH7LISWCXQWOAVCBSULQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507055783 (87h26m45.378138818s) [noise=true , address=145.79.227.41:4086 , piece_id=GXH4GU22OG3BZQSHTK25AHWPFCYKCQEN72NXEPYXO4ZUIW2RSDEA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507493356 (87h26m45.378576391s) [noise=true , address=145.79.227.29:4942 , piece_id=SMZGQG7OUYGCHPPJLIS3PH5Y3VU65I46D4NYL2PXCBRSPQRSPZEQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507543853 (87h26m45.378626898s) [noise=true , address=145.79.227.16:4909 , piece_id=7TX6RV44XTP4C5DPVEPTSV6ECNGUSXU6G33T6L3AUCFNQLISNHOQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507799740 (87h26m45.378882775s) [noise=true , address=145.79.227.29:4941 , piece_id=UDD375EXXARYRWOXMWZF25HB7VW4YVWQEFESPW4JNNB4MG7QPPSA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507920349 (87h26m45.379003384s) [noise=true , address=145.79.227.31:4797 , piece_id=AW3Q7GXNWAU5T5Z562KBKXYZPJEL5IRR6XB7EZZPZXUJMBQNOSBA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508339767 (87h26m45.379422802s) [noise=true , address=145.79.227.34:4905 , piece_id=RUHC4SR4EO2IF2KEDJNYZOY33EAKF4JR4FTM62NZJCUD2XPO7N6Q, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508566930 (87h26m45.379649965s) [noise=true , address=145.79.227.25:4618 , piece_id=K2Y25ECEZ6SEYOV5NPG2EEWFVN5IFFVM54QPK4ZGWZQRNE3I3BCA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508739269 (87h26m45.379822314s) [noise=true , address=145.79.227.24:4581 , piece_id=WNL2FDFC4N4U5TX522OSVZDEBNO6VVZTL4TOS2C65KHY2CAMHQHA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509134110 (87h26m45.380217145s) [noise=true , address=145.79.227.24:4582 , piece_id=EAMAU5U3OBF5RA762ZYOARJZ4J2J4MXNEKWKY3JWGT3BX76OB6XQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509258507 (87h26m45.380341543s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509546506 (87h26m45.380629541s) [noise=true , address=145.79.227.21:4473 , piece_id=E4G6ZYPNOHCKQ6FYK4I3Y5V36C7RX5TOKVCQKN2VKHFWV5J62VEA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509843851 (87h26m45.380926896s) [noise=true , address=145.79.227.23:4547 , piece_id=UDL6VDEHGJ6Z4HMRU6HM6XIDUCMSWCEECCUH45FT3FQVLUDOEZGQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.510141709 (87h26m45.381224754s) [noise=true , address=145.79.227.38:4977 , piece_id=JZVULFGKR65GE24SFP5U7LHINOZIZ2BXIGQSBTZUSPXEWXEQA32A, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.510428084 (87h26m45.381511129s) [noise=true , address=145.79.227.32:4834 , piece_id=N6VJKJMTE5W3HWGCXDGPTLB3BHSFLWBMESU57UOSWPNBM55DZ4MQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:46.567032201 (87h26m46.438115236s) [noise=true , address=77.90.31.15:4698 , piece_id=2Y3GG5Y2QUJ6PFAYODA4V63N4CQXCTLRNH7LISWCXQWOAVCBSULQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 06:55:47.200741008 (87h26m47.071824053s) [upload=76331]: Next returned nil segment Dec 9 06:55:47.200756667 (87h26m47.071839702s) [upload=76331]: Waiting for error group managing segments... Dec 9 06:55:47.488289311 (87h26m47.359372346s) [noise=true , address=145.79.227.41:4086 , piece_id=GXH4GU22OG3BZQSHTK25AHWPFCYKCQEN72NXEPYXO4ZUIW2RSDEA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.488813710 (87h26m47.359896755s) [noise=true , address=145.79.227.21:4473 , piece_id=E4G6ZYPNOHCKQ6FYK4I3Y5V36C7RX5TOKVCQKN2VKHFWV5J62VEA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.488891818 (87h26m47.359974853s) [noise=true , address=145.79.227.38:4977 , piece_id=JZVULFGKR65GE24SFP5U7LHINOZIZ2BXIGQSBTZUSPXEWXEQA32A, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.489842157 (87h26m47.360925202s) [noise=true , address=145.79.227.24:4582 , piece_id=EAMAU5U3OBF5RA762ZYOARJZ4J2J4MXNEKWKY3JWGT3BX76OB6XQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.489996712 (87h26m47.361079747s) [noise=true , address=145.79.227.32:4834 , piece_id=N6VJKJMTE5W3HWGCXDGPTLB3BHSFLWBMESU57UOSWPNBM55DZ4MQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.495771257 (87h26m47.366854302s) [noise=true , address=145.79.227.34:4905 , piece_id=RUHC4SR4EO2IF2KEDJNYZOY33EAKF4JR4FTM62NZJCUD2XPO7N6Q, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.496288763 (87h26m47.367371808s) [noise=true , address=145.79.227.25:4618 , piece_id=K2Y25ECEZ6SEYOV5NPG2EEWFVN5IFFVM54QPK4ZGWZQRNE3I3BCA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.500744707 (87h26m47.371827752s) [noise=true , address=145.79.227.31:4797 , piece_id=AW3Q7GXNWAU5T5Z562KBKXYZPJEL5IRR6XB7EZZPZXUJMBQNOSBA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.501416546 (87h26m47.372499591s) [noise=true , address=145.79.227.16:4909 , piece_id=7TX6RV44XTP4C5DPVEPTSV6ECNGUSXU6G33T6L3AUCFNQLISNHOQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.501422417 (87h26m47.372505462s) [noise=true , address=145.79.227.23:4547 , piece_id=UDL6VDEHGJ6Z4HMRU6HM6XIDUCMSWCEECCUH45FT3FQVLUDOEZGQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.514960194 (87h26m47.386043249s) [noise=true , address=145.79.227.24:4581 , piece_id=WNL2FDFC4N4U5TX522OSVZDEBNO6VVZTL4TOS2C65KHY2CAMHQHA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516209051 (87h26m47.387292086s) [noise=true , address=145.79.227.29:4941 , piece_id=UDD375EXXARYRWOXMWZF25HB7VW4YVWQEFESPW4JNNB4MG7QPPSA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516598713 (87h26m47.387681758s) [noise=true , address=145.79.227.29:4942 , piece_id=SMZGQG7OUYGCHPPJLIS3PH5Y3VU65I46D4NYL2PXCBRSPQRSPZEQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516607911 (87h26m47.387690946s) [seg_pos={0 0}, upload=76331]: Segment reached optimal threshold of 13 pieces. Dec 9 06:55:47.516785880 (87h26m47.387868915s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u): protocol: expected piece hash; context canceled; EOF Dec 9 06:55:47.516824383 (87h26m47.387907448s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Long tail cancellation due to optimal threshold Dec 9 06:55:47.517129214 (87h26m47.388212259s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 06:55:47.517138261 (87h26m47.388221306s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Long tail cancellation due to optimal threshold Dec 9 06:55:47.517375302 (87h26m47.388458337s) [seg_pos={0 0}, upload=76331]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 06:55:47.517466135 (87h26m47.38854917s) [upload=76331]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 06:55:47.517473488 (87h26m47.388556534s) [upload=76331]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:00:17.830318594 (87h31m17.701401639s) [upload=76331]: Done uploading segments... Dec 9 07:00:17.830402994 (87h31m17.70148605s) [upload=76331]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.724632722s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.113846494s trace: https://jaeger.dev.storj.io/trace/488f36bcf1592ab1 FAIL (2025-12-09 07:20:20 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.113879247s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.X................X...X...X............................X....] duration: 6.079333584s trace: https://jaeger.dev.storj.io/trace/2225ff488c04e76d FAIL (2025-12-09 07:20:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.079365265s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........X....X...........................................] duration: 3.53824478s trace: https://jaeger.dev.storj.io/trace/2c78734d9bae1209 FAIL (2025-12-09 07:20:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.538280558s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.010322436s trace: https://jaeger.dev.storj.io/trace/488f36bcf1592ab1 FAIL (2025-12-09 07:19:27 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.010352465s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.570725095s trace: https://jaeger.dev.storj.io/trace/3a1fb019f7c2965d FAIL (2025-12-09 07:19:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 06:55:45.259556946 (87h26m45.130639981s) [upload=76331]: Starting upload Dec 9 06:55:45.259623222 (87h26m45.130706257s) [upload=76331]: Done starting upload Dec 9 06:55:45.259658870 (87h26m45.130741905s) [upload=76331]: Uploading segments... Dec 9 06:55:45.259795500 (87h26m45.130878535s) [upload=76331]: Got next segment. Inline: false Dec 9 06:55:45.259821169 (87h26m45.130904204s) [upload=76331]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 06:55:45.259825137 (87h26m45.130908172s) [upload=76331]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 06:55:45.506331314 (87h26m45.377414359s) [seg_pos={0 0}, upload=76331]: Begin upload segment... Dec 9 06:55:45.506397410 (87h26m45.377480445s) [seg_pos={0 0}, upload=76331]: Done begin upload segment. Dec 9 06:55:45.506452925 (87h26m45.37753596s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.506473765 (87h26m45.3775568s) [noise=true , address=77.90.31.15:4698 , piece_id=2Y3GG5Y2QUJ6PFAYODA4V63N4CQXCTLRNH7LISWCXQWOAVCBSULQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507055783 (87h26m45.378138818s) [noise=true , address=145.79.227.41:4086 , piece_id=GXH4GU22OG3BZQSHTK25AHWPFCYKCQEN72NXEPYXO4ZUIW2RSDEA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507493356 (87h26m45.378576391s) [noise=true , address=145.79.227.29:4942 , piece_id=SMZGQG7OUYGCHPPJLIS3PH5Y3VU65I46D4NYL2PXCBRSPQRSPZEQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507543853 (87h26m45.378626898s) [noise=true , address=145.79.227.16:4909 , piece_id=7TX6RV44XTP4C5DPVEPTSV6ECNGUSXU6G33T6L3AUCFNQLISNHOQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507799740 (87h26m45.378882775s) [noise=true , address=145.79.227.29:4941 , piece_id=UDD375EXXARYRWOXMWZF25HB7VW4YVWQEFESPW4JNNB4MG7QPPSA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507920349 (87h26m45.379003384s) [noise=true , address=145.79.227.31:4797 , piece_id=AW3Q7GXNWAU5T5Z562KBKXYZPJEL5IRR6XB7EZZPZXUJMBQNOSBA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508339767 (87h26m45.379422802s) [noise=true , address=145.79.227.34:4905 , piece_id=RUHC4SR4EO2IF2KEDJNYZOY33EAKF4JR4FTM62NZJCUD2XPO7N6Q, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508566930 (87h26m45.379649965s) [noise=true , address=145.79.227.25:4618 , piece_id=K2Y25ECEZ6SEYOV5NPG2EEWFVN5IFFVM54QPK4ZGWZQRNE3I3BCA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508739269 (87h26m45.379822314s) [noise=true , address=145.79.227.24:4581 , piece_id=WNL2FDFC4N4U5TX522OSVZDEBNO6VVZTL4TOS2C65KHY2CAMHQHA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509134110 (87h26m45.380217145s) [noise=true , address=145.79.227.24:4582 , piece_id=EAMAU5U3OBF5RA762ZYOARJZ4J2J4MXNEKWKY3JWGT3BX76OB6XQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509258507 (87h26m45.380341543s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509546506 (87h26m45.380629541s) [noise=true , address=145.79.227.21:4473 , piece_id=E4G6ZYPNOHCKQ6FYK4I3Y5V36C7RX5TOKVCQKN2VKHFWV5J62VEA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509843851 (87h26m45.380926896s) [noise=true , address=145.79.227.23:4547 , piece_id=UDL6VDEHGJ6Z4HMRU6HM6XIDUCMSWCEECCUH45FT3FQVLUDOEZGQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.510141709 (87h26m45.381224754s) [noise=true , address=145.79.227.38:4977 , piece_id=JZVULFGKR65GE24SFP5U7LHINOZIZ2BXIGQSBTZUSPXEWXEQA32A, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.510428084 (87h26m45.381511129s) [noise=true , address=145.79.227.32:4834 , piece_id=N6VJKJMTE5W3HWGCXDGPTLB3BHSFLWBMESU57UOSWPNBM55DZ4MQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:46.567032201 (87h26m46.438115236s) [noise=true , address=77.90.31.15:4698 , piece_id=2Y3GG5Y2QUJ6PFAYODA4V63N4CQXCTLRNH7LISWCXQWOAVCBSULQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 06:55:47.200741008 (87h26m47.071824053s) [upload=76331]: Next returned nil segment Dec 9 06:55:47.200756667 (87h26m47.071839702s) [upload=76331]: Waiting for error group managing segments... Dec 9 06:55:47.488289311 (87h26m47.359372346s) [noise=true , address=145.79.227.41:4086 , piece_id=GXH4GU22OG3BZQSHTK25AHWPFCYKCQEN72NXEPYXO4ZUIW2RSDEA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.488813710 (87h26m47.359896755s) [noise=true , address=145.79.227.21:4473 , piece_id=E4G6ZYPNOHCKQ6FYK4I3Y5V36C7RX5TOKVCQKN2VKHFWV5J62VEA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.488891818 (87h26m47.359974853s) [noise=true , address=145.79.227.38:4977 , piece_id=JZVULFGKR65GE24SFP5U7LHINOZIZ2BXIGQSBTZUSPXEWXEQA32A, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.489842157 (87h26m47.360925202s) [noise=true , address=145.79.227.24:4582 , piece_id=EAMAU5U3OBF5RA762ZYOARJZ4J2J4MXNEKWKY3JWGT3BX76OB6XQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.489996712 (87h26m47.361079747s) [noise=true , address=145.79.227.32:4834 , piece_id=N6VJKJMTE5W3HWGCXDGPTLB3BHSFLWBMESU57UOSWPNBM55DZ4MQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.495771257 (87h26m47.366854302s) [noise=true , address=145.79.227.34:4905 , piece_id=RUHC4SR4EO2IF2KEDJNYZOY33EAKF4JR4FTM62NZJCUD2XPO7N6Q, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.496288763 (87h26m47.367371808s) [noise=true , address=145.79.227.25:4618 , piece_id=K2Y25ECEZ6SEYOV5NPG2EEWFVN5IFFVM54QPK4ZGWZQRNE3I3BCA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.500744707 (87h26m47.371827752s) [noise=true , address=145.79.227.31:4797 , piece_id=AW3Q7GXNWAU5T5Z562KBKXYZPJEL5IRR6XB7EZZPZXUJMBQNOSBA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.501416546 (87h26m47.372499591s) [noise=true , address=145.79.227.16:4909 , piece_id=7TX6RV44XTP4C5DPVEPTSV6ECNGUSXU6G33T6L3AUCFNQLISNHOQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.501422417 (87h26m47.372505462s) [noise=true , address=145.79.227.23:4547 , piece_id=UDL6VDEHGJ6Z4HMRU6HM6XIDUCMSWCEECCUH45FT3FQVLUDOEZGQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.514960194 (87h26m47.386043249s) [noise=true , address=145.79.227.24:4581 , piece_id=WNL2FDFC4N4U5TX522OSVZDEBNO6VVZTL4TOS2C65KHY2CAMHQHA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516209051 (87h26m47.387292086s) [noise=true , address=145.79.227.29:4941 , piece_id=UDD375EXXARYRWOXMWZF25HB7VW4YVWQEFESPW4JNNB4MG7QPPSA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516598713 (87h26m47.387681758s) [noise=true , address=145.79.227.29:4942 , piece_id=SMZGQG7OUYGCHPPJLIS3PH5Y3VU65I46D4NYL2PXCBRSPQRSPZEQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516607911 (87h26m47.387690946s) [seg_pos={0 0}, upload=76331]: Segment reached optimal threshold of 13 pieces. Dec 9 06:55:47.516785880 (87h26m47.387868915s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u): protocol: expected piece hash; context canceled; EOF Dec 9 06:55:47.516824383 (87h26m47.387907448s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Long tail cancellation due to optimal threshold Dec 9 06:55:47.517129214 (87h26m47.388212259s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 06:55:47.517138261 (87h26m47.388221306s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Long tail cancellation due to optimal threshold Dec 9 06:55:47.517375302 (87h26m47.388458337s) [seg_pos={0 0}, upload=76331]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 06:55:47.517466135 (87h26m47.38854917s) [upload=76331]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 06:55:47.517473488 (87h26m47.388556534s) [upload=76331]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:00:17.830318594 (87h31m17.701401639s) [upload=76331]: Done uploading segments... Dec 9 07:00:17.830402994 (87h31m17.70148605s) [upload=76331]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.570754912s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............X....X..........................................] duration: 3.492320204s trace: https://jaeger.dev.storj.io/trace/2c78734d9bae1209 FAIL (2025-12-09 07:19:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.492362573s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..X................X...X...X............................X...] duration: 5.930504778s trace: https://jaeger.dev.storj.io/trace/2225ff488c04e76d FAIL (2025-12-09 07:19:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.93058957s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............X....X.........................................] duration: 3.470496844s trace: https://jaeger.dev.storj.io/trace/2c78734d9bae1209 FAIL (2025-12-09 07:18:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.470525669s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.155994553s trace: https://jaeger.dev.storj.io/trace/488f36bcf1592ab1 FAIL (2025-12-09 07:18:25 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.156019561s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............X....X........................................] duration: 3.45419745s trace: https://jaeger.dev.storj.io/trace/2c78734d9bae1209 FAIL (2025-12-09 07:18:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.454390868s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.716752387s trace: https://jaeger.dev.storj.io/trace/3a1fb019f7c2965d FAIL (2025-12-09 07:18:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 06:55:45.259556946 (87h26m45.130639981s) [upload=76331]: Starting upload Dec 9 06:55:45.259623222 (87h26m45.130706257s) [upload=76331]: Done starting upload Dec 9 06:55:45.259658870 (87h26m45.130741905s) [upload=76331]: Uploading segments... Dec 9 06:55:45.259795500 (87h26m45.130878535s) [upload=76331]: Got next segment. Inline: false Dec 9 06:55:45.259821169 (87h26m45.130904204s) [upload=76331]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 06:55:45.259825137 (87h26m45.130908172s) [upload=76331]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 06:55:45.506331314 (87h26m45.377414359s) [seg_pos={0 0}, upload=76331]: Begin upload segment... Dec 9 06:55:45.506397410 (87h26m45.377480445s) [seg_pos={0 0}, upload=76331]: Done begin upload segment. Dec 9 06:55:45.506452925 (87h26m45.37753596s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.506473765 (87h26m45.3775568s) [noise=true , address=77.90.31.15:4698 , piece_id=2Y3GG5Y2QUJ6PFAYODA4V63N4CQXCTLRNH7LISWCXQWOAVCBSULQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507055783 (87h26m45.378138818s) [noise=true , address=145.79.227.41:4086 , piece_id=GXH4GU22OG3BZQSHTK25AHWPFCYKCQEN72NXEPYXO4ZUIW2RSDEA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507493356 (87h26m45.378576391s) [noise=true , address=145.79.227.29:4942 , piece_id=SMZGQG7OUYGCHPPJLIS3PH5Y3VU65I46D4NYL2PXCBRSPQRSPZEQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507543853 (87h26m45.378626898s) [noise=true , address=145.79.227.16:4909 , piece_id=7TX6RV44XTP4C5DPVEPTSV6ECNGUSXU6G33T6L3AUCFNQLISNHOQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507799740 (87h26m45.378882775s) [noise=true , address=145.79.227.29:4941 , piece_id=UDD375EXXARYRWOXMWZF25HB7VW4YVWQEFESPW4JNNB4MG7QPPSA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507920349 (87h26m45.379003384s) [noise=true , address=145.79.227.31:4797 , piece_id=AW3Q7GXNWAU5T5Z562KBKXYZPJEL5IRR6XB7EZZPZXUJMBQNOSBA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508339767 (87h26m45.379422802s) [noise=true , address=145.79.227.34:4905 , piece_id=RUHC4SR4EO2IF2KEDJNYZOY33EAKF4JR4FTM62NZJCUD2XPO7N6Q, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508566930 (87h26m45.379649965s) [noise=true , address=145.79.227.25:4618 , piece_id=K2Y25ECEZ6SEYOV5NPG2EEWFVN5IFFVM54QPK4ZGWZQRNE3I3BCA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508739269 (87h26m45.379822314s) [noise=true , address=145.79.227.24:4581 , piece_id=WNL2FDFC4N4U5TX522OSVZDEBNO6VVZTL4TOS2C65KHY2CAMHQHA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509134110 (87h26m45.380217145s) [noise=true , address=145.79.227.24:4582 , piece_id=EAMAU5U3OBF5RA762ZYOARJZ4J2J4MXNEKWKY3JWGT3BX76OB6XQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509258507 (87h26m45.380341543s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509546506 (87h26m45.380629541s) [noise=true , address=145.79.227.21:4473 , piece_id=E4G6ZYPNOHCKQ6FYK4I3Y5V36C7RX5TOKVCQKN2VKHFWV5J62VEA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509843851 (87h26m45.380926896s) [noise=true , address=145.79.227.23:4547 , piece_id=UDL6VDEHGJ6Z4HMRU6HM6XIDUCMSWCEECCUH45FT3FQVLUDOEZGQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.510141709 (87h26m45.381224754s) [noise=true , address=145.79.227.38:4977 , piece_id=JZVULFGKR65GE24SFP5U7LHINOZIZ2BXIGQSBTZUSPXEWXEQA32A, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.510428084 (87h26m45.381511129s) [noise=true , address=145.79.227.32:4834 , piece_id=N6VJKJMTE5W3HWGCXDGPTLB3BHSFLWBMESU57UOSWPNBM55DZ4MQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:46.567032201 (87h26m46.438115236s) [noise=true , address=77.90.31.15:4698 , piece_id=2Y3GG5Y2QUJ6PFAYODA4V63N4CQXCTLRNH7LISWCXQWOAVCBSULQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 06:55:47.200741008 (87h26m47.071824053s) [upload=76331]: Next returned nil segment Dec 9 06:55:47.200756667 (87h26m47.071839702s) [upload=76331]: Waiting for error group managing segments... Dec 9 06:55:47.488289311 (87h26m47.359372346s) [noise=true , address=145.79.227.41:4086 , piece_id=GXH4GU22OG3BZQSHTK25AHWPFCYKCQEN72NXEPYXO4ZUIW2RSDEA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.488813710 (87h26m47.359896755s) [noise=true , address=145.79.227.21:4473 , piece_id=E4G6ZYPNOHCKQ6FYK4I3Y5V36C7RX5TOKVCQKN2VKHFWV5J62VEA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.488891818 (87h26m47.359974853s) [noise=true , address=145.79.227.38:4977 , piece_id=JZVULFGKR65GE24SFP5U7LHINOZIZ2BXIGQSBTZUSPXEWXEQA32A, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.489842157 (87h26m47.360925202s) [noise=true , address=145.79.227.24:4582 , piece_id=EAMAU5U3OBF5RA762ZYOARJZ4J2J4MXNEKWKY3JWGT3BX76OB6XQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.489996712 (87h26m47.361079747s) [noise=true , address=145.79.227.32:4834 , piece_id=N6VJKJMTE5W3HWGCXDGPTLB3BHSFLWBMESU57UOSWPNBM55DZ4MQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.495771257 (87h26m47.366854302s) [noise=true , address=145.79.227.34:4905 , piece_id=RUHC4SR4EO2IF2KEDJNYZOY33EAKF4JR4FTM62NZJCUD2XPO7N6Q, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.496288763 (87h26m47.367371808s) [noise=true , address=145.79.227.25:4618 , piece_id=K2Y25ECEZ6SEYOV5NPG2EEWFVN5IFFVM54QPK4ZGWZQRNE3I3BCA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.500744707 (87h26m47.371827752s) [noise=true , address=145.79.227.31:4797 , piece_id=AW3Q7GXNWAU5T5Z562KBKXYZPJEL5IRR6XB7EZZPZXUJMBQNOSBA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.501416546 (87h26m47.372499591s) [noise=true , address=145.79.227.16:4909 , piece_id=7TX6RV44XTP4C5DPVEPTSV6ECNGUSXU6G33T6L3AUCFNQLISNHOQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.501422417 (87h26m47.372505462s) [noise=true , address=145.79.227.23:4547 , piece_id=UDL6VDEHGJ6Z4HMRU6HM6XIDUCMSWCEECCUH45FT3FQVLUDOEZGQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.514960194 (87h26m47.386043249s) [noise=true , address=145.79.227.24:4581 , piece_id=WNL2FDFC4N4U5TX522OSVZDEBNO6VVZTL4TOS2C65KHY2CAMHQHA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516209051 (87h26m47.387292086s) [noise=true , address=145.79.227.29:4941 , piece_id=UDD375EXXARYRWOXMWZF25HB7VW4YVWQEFESPW4JNNB4MG7QPPSA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516598713 (87h26m47.387681758s) [noise=true , address=145.79.227.29:4942 , piece_id=SMZGQG7OUYGCHPPJLIS3PH5Y3VU65I46D4NYL2PXCBRSPQRSPZEQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516607911 (87h26m47.387690946s) [seg_pos={0 0}, upload=76331]: Segment reached optimal threshold of 13 pieces. Dec 9 06:55:47.516785880 (87h26m47.387868915s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u): protocol: expected piece hash; context canceled; EOF Dec 9 06:55:47.516824383 (87h26m47.387907448s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Long tail cancellation due to optimal threshold Dec 9 06:55:47.517129214 (87h26m47.388212259s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 06:55:47.517138261 (87h26m47.388221306s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Long tail cancellation due to optimal threshold Dec 9 06:55:47.517375302 (87h26m47.388458337s) [seg_pos={0 0}, upload=76331]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 06:55:47.517466135 (87h26m47.38854917s) [upload=76331]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 06:55:47.517473488 (87h26m47.388556534s) [upload=76331]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:00:17.830318594 (87h31m17.701401639s) [upload=76331]: Done uploading segments... Dec 9 07:00:17.830402994 (87h31m17.70148605s) [upload=76331]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.71678059s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 4.978324466s trace: https://jaeger.dev.storj.io/trace/488f36bcf1592ab1 FAIL (2025-12-09 07:18:03 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.978360584s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............X....X.......................................] duration: 3.407422629s trace: https://jaeger.dev.storj.io/trace/2c78734d9bae1209 FAIL (2025-12-09 07:17:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.407452225s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [...X................X...X...X............................X..] duration: 5.788600717s trace: https://jaeger.dev.storj.io/trace/2225ff488c04e76d FAIL (2025-12-09 07:17:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.788624842s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [....X................X...X...X............................X.] duration: 5.652031866s trace: https://jaeger.dev.storj.io/trace/2225ff488c04e76d FAIL (2025-12-09 07:17:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.652064037s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 4.835787384s trace: https://jaeger.dev.storj.io/trace/488f36bcf1592ab1 FAIL (2025-12-09 07:17:25 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.835819936s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.....X................X...X...X............................X] duration: 1m0.000669615s trace: https://jaeger.dev.storj.io/trace/2225ff488c04e76d FAIL (2025-12-09 07:17:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.00068276s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.565339639s trace: https://jaeger.dev.storj.io/trace/3a1fb019f7c2965d FAIL (2025-12-09 07:17:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 06:55:45.259556946 (87h26m45.130639981s) [upload=76331]: Starting upload Dec 9 06:55:45.259623222 (87h26m45.130706257s) [upload=76331]: Done starting upload Dec 9 06:55:45.259658870 (87h26m45.130741905s) [upload=76331]: Uploading segments... Dec 9 06:55:45.259795500 (87h26m45.130878535s) [upload=76331]: Got next segment. Inline: false Dec 9 06:55:45.259821169 (87h26m45.130904204s) [upload=76331]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 06:55:45.259825137 (87h26m45.130908172s) [upload=76331]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 06:55:45.506331314 (87h26m45.377414359s) [seg_pos={0 0}, upload=76331]: Begin upload segment... Dec 9 06:55:45.506397410 (87h26m45.377480445s) [seg_pos={0 0}, upload=76331]: Done begin upload segment. Dec 9 06:55:45.506452925 (87h26m45.37753596s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.506473765 (87h26m45.3775568s) [noise=true , address=77.90.31.15:4698 , piece_id=2Y3GG5Y2QUJ6PFAYODA4V63N4CQXCTLRNH7LISWCXQWOAVCBSULQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507055783 (87h26m45.378138818s) [noise=true , address=145.79.227.41:4086 , piece_id=GXH4GU22OG3BZQSHTK25AHWPFCYKCQEN72NXEPYXO4ZUIW2RSDEA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507493356 (87h26m45.378576391s) [noise=true , address=145.79.227.29:4942 , piece_id=SMZGQG7OUYGCHPPJLIS3PH5Y3VU65I46D4NYL2PXCBRSPQRSPZEQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507543853 (87h26m45.378626898s) [noise=true , address=145.79.227.16:4909 , piece_id=7TX6RV44XTP4C5DPVEPTSV6ECNGUSXU6G33T6L3AUCFNQLISNHOQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507799740 (87h26m45.378882775s) [noise=true , address=145.79.227.29:4941 , piece_id=UDD375EXXARYRWOXMWZF25HB7VW4YVWQEFESPW4JNNB4MG7QPPSA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507920349 (87h26m45.379003384s) [noise=true , address=145.79.227.31:4797 , piece_id=AW3Q7GXNWAU5T5Z562KBKXYZPJEL5IRR6XB7EZZPZXUJMBQNOSBA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508339767 (87h26m45.379422802s) [noise=true , address=145.79.227.34:4905 , piece_id=RUHC4SR4EO2IF2KEDJNYZOY33EAKF4JR4FTM62NZJCUD2XPO7N6Q, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508566930 (87h26m45.379649965s) [noise=true , address=145.79.227.25:4618 , piece_id=K2Y25ECEZ6SEYOV5NPG2EEWFVN5IFFVM54QPK4ZGWZQRNE3I3BCA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508739269 (87h26m45.379822314s) [noise=true , address=145.79.227.24:4581 , piece_id=WNL2FDFC4N4U5TX522OSVZDEBNO6VVZTL4TOS2C65KHY2CAMHQHA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509134110 (87h26m45.380217145s) [noise=true , address=145.79.227.24:4582 , piece_id=EAMAU5U3OBF5RA762ZYOARJZ4J2J4MXNEKWKY3JWGT3BX76OB6XQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509258507 (87h26m45.380341543s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509546506 (87h26m45.380629541s) [noise=true , address=145.79.227.21:4473 , piece_id=E4G6ZYPNOHCKQ6FYK4I3Y5V36C7RX5TOKVCQKN2VKHFWV5J62VEA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509843851 (87h26m45.380926896s) [noise=true , address=145.79.227.23:4547 , piece_id=UDL6VDEHGJ6Z4HMRU6HM6XIDUCMSWCEECCUH45FT3FQVLUDOEZGQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.510141709 (87h26m45.381224754s) [noise=true , address=145.79.227.38:4977 , piece_id=JZVULFGKR65GE24SFP5U7LHINOZIZ2BXIGQSBTZUSPXEWXEQA32A, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.510428084 (87h26m45.381511129s) [noise=true , address=145.79.227.32:4834 , piece_id=N6VJKJMTE5W3HWGCXDGPTLB3BHSFLWBMESU57UOSWPNBM55DZ4MQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:46.567032201 (87h26m46.438115236s) [noise=true , address=77.90.31.15:4698 , piece_id=2Y3GG5Y2QUJ6PFAYODA4V63N4CQXCTLRNH7LISWCXQWOAVCBSULQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 06:55:47.200741008 (87h26m47.071824053s) [upload=76331]: Next returned nil segment Dec 9 06:55:47.200756667 (87h26m47.071839702s) [upload=76331]: Waiting for error group managing segments... Dec 9 06:55:47.488289311 (87h26m47.359372346s) [noise=true , address=145.79.227.41:4086 , piece_id=GXH4GU22OG3BZQSHTK25AHWPFCYKCQEN72NXEPYXO4ZUIW2RSDEA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.488813710 (87h26m47.359896755s) [noise=true , address=145.79.227.21:4473 , piece_id=E4G6ZYPNOHCKQ6FYK4I3Y5V36C7RX5TOKVCQKN2VKHFWV5J62VEA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.488891818 (87h26m47.359974853s) [noise=true , address=145.79.227.38:4977 , piece_id=JZVULFGKR65GE24SFP5U7LHINOZIZ2BXIGQSBTZUSPXEWXEQA32A, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.489842157 (87h26m47.360925202s) [noise=true , address=145.79.227.24:4582 , piece_id=EAMAU5U3OBF5RA762ZYOARJZ4J2J4MXNEKWKY3JWGT3BX76OB6XQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.489996712 (87h26m47.361079747s) [noise=true , address=145.79.227.32:4834 , piece_id=N6VJKJMTE5W3HWGCXDGPTLB3BHSFLWBMESU57UOSWPNBM55DZ4MQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.495771257 (87h26m47.366854302s) [noise=true , address=145.79.227.34:4905 , piece_id=RUHC4SR4EO2IF2KEDJNYZOY33EAKF4JR4FTM62NZJCUD2XPO7N6Q, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.496288763 (87h26m47.367371808s) [noise=true , address=145.79.227.25:4618 , piece_id=K2Y25ECEZ6SEYOV5NPG2EEWFVN5IFFVM54QPK4ZGWZQRNE3I3BCA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.500744707 (87h26m47.371827752s) [noise=true , address=145.79.227.31:4797 , piece_id=AW3Q7GXNWAU5T5Z562KBKXYZPJEL5IRR6XB7EZZPZXUJMBQNOSBA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.501416546 (87h26m47.372499591s) [noise=true , address=145.79.227.16:4909 , piece_id=7TX6RV44XTP4C5DPVEPTSV6ECNGUSXU6G33T6L3AUCFNQLISNHOQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.501422417 (87h26m47.372505462s) [noise=true , address=145.79.227.23:4547 , piece_id=UDL6VDEHGJ6Z4HMRU6HM6XIDUCMSWCEECCUH45FT3FQVLUDOEZGQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.514960194 (87h26m47.386043249s) [noise=true , address=145.79.227.24:4581 , piece_id=WNL2FDFC4N4U5TX522OSVZDEBNO6VVZTL4TOS2C65KHY2CAMHQHA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516209051 (87h26m47.387292086s) [noise=true , address=145.79.227.29:4941 , piece_id=UDD375EXXARYRWOXMWZF25HB7VW4YVWQEFESPW4JNNB4MG7QPPSA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516598713 (87h26m47.387681758s) [noise=true , address=145.79.227.29:4942 , piece_id=SMZGQG7OUYGCHPPJLIS3PH5Y3VU65I46D4NYL2PXCBRSPQRSPZEQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516607911 (87h26m47.387690946s) [seg_pos={0 0}, upload=76331]: Segment reached optimal threshold of 13 pieces. Dec 9 06:55:47.516785880 (87h26m47.387868915s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u): protocol: expected piece hash; context canceled; EOF Dec 9 06:55:47.516824383 (87h26m47.387907448s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Long tail cancellation due to optimal threshold Dec 9 06:55:47.517129214 (87h26m47.388212259s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 06:55:47.517138261 (87h26m47.388221306s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Long tail cancellation due to optimal threshold Dec 9 06:55:47.517375302 (87h26m47.388458337s) [seg_pos={0 0}, upload=76331]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 06:55:47.517466135 (87h26m47.38854917s) [upload=76331]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 06:55:47.517473488 (87h26m47.388556534s) [upload=76331]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:00:17.830318594 (87h31m17.701401639s) [upload=76331]: Done uploading segments... Dec 9 07:00:17.830402994 (87h31m17.70148605s) [upload=76331]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.565371048s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [X...............X....X......................................] duration: 3.349359366s trace: https://jaeger.dev.storj.io/trace/2c78734d9bae1209 FAIL (2025-12-09 07:17:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.349466542s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.072179364s trace: https://jaeger.dev.storj.io/trace/488f36bcf1592ab1 FAIL (2025-12-09 07:16:54 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.072211105s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.X...............X....X.....................................] duration: 3.474309545s trace: https://jaeger.dev.storj.io/trace/2c78734d9bae1209 FAIL (2025-12-09 07:16:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.474378026s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.672877664s trace: https://jaeger.dev.storj.io/trace/3a1fb019f7c2965d FAIL (2025-12-09 07:16:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 06:55:45.259556946 (87h26m45.130639981s) [upload=76331]: Starting upload Dec 9 06:55:45.259623222 (87h26m45.130706257s) [upload=76331]: Done starting upload Dec 9 06:55:45.259658870 (87h26m45.130741905s) [upload=76331]: Uploading segments... Dec 9 06:55:45.259795500 (87h26m45.130878535s) [upload=76331]: Got next segment. Inline: false Dec 9 06:55:45.259821169 (87h26m45.130904204s) [upload=76331]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 06:55:45.259825137 (87h26m45.130908172s) [upload=76331]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 06:55:45.506331314 (87h26m45.377414359s) [seg_pos={0 0}, upload=76331]: Begin upload segment... Dec 9 06:55:45.506397410 (87h26m45.377480445s) [seg_pos={0 0}, upload=76331]: Done begin upload segment. Dec 9 06:55:45.506452925 (87h26m45.37753596s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.506473765 (87h26m45.3775568s) [noise=true , address=77.90.31.15:4698 , piece_id=2Y3GG5Y2QUJ6PFAYODA4V63N4CQXCTLRNH7LISWCXQWOAVCBSULQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507055783 (87h26m45.378138818s) [noise=true , address=145.79.227.41:4086 , piece_id=GXH4GU22OG3BZQSHTK25AHWPFCYKCQEN72NXEPYXO4ZUIW2RSDEA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507493356 (87h26m45.378576391s) [noise=true , address=145.79.227.29:4942 , piece_id=SMZGQG7OUYGCHPPJLIS3PH5Y3VU65I46D4NYL2PXCBRSPQRSPZEQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507543853 (87h26m45.378626898s) [noise=true , address=145.79.227.16:4909 , piece_id=7TX6RV44XTP4C5DPVEPTSV6ECNGUSXU6G33T6L3AUCFNQLISNHOQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507799740 (87h26m45.378882775s) [noise=true , address=145.79.227.29:4941 , piece_id=UDD375EXXARYRWOXMWZF25HB7VW4YVWQEFESPW4JNNB4MG7QPPSA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507920349 (87h26m45.379003384s) [noise=true , address=145.79.227.31:4797 , piece_id=AW3Q7GXNWAU5T5Z562KBKXYZPJEL5IRR6XB7EZZPZXUJMBQNOSBA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508339767 (87h26m45.379422802s) [noise=true , address=145.79.227.34:4905 , piece_id=RUHC4SR4EO2IF2KEDJNYZOY33EAKF4JR4FTM62NZJCUD2XPO7N6Q, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508566930 (87h26m45.379649965s) [noise=true , address=145.79.227.25:4618 , piece_id=K2Y25ECEZ6SEYOV5NPG2EEWFVN5IFFVM54QPK4ZGWZQRNE3I3BCA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508739269 (87h26m45.379822314s) [noise=true , address=145.79.227.24:4581 , piece_id=WNL2FDFC4N4U5TX522OSVZDEBNO6VVZTL4TOS2C65KHY2CAMHQHA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509134110 (87h26m45.380217145s) [noise=true , address=145.79.227.24:4582 , piece_id=EAMAU5U3OBF5RA762ZYOARJZ4J2J4MXNEKWKY3JWGT3BX76OB6XQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509258507 (87h26m45.380341543s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509546506 (87h26m45.380629541s) [noise=true , address=145.79.227.21:4473 , piece_id=E4G6ZYPNOHCKQ6FYK4I3Y5V36C7RX5TOKVCQKN2VKHFWV5J62VEA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509843851 (87h26m45.380926896s) [noise=true , address=145.79.227.23:4547 , piece_id=UDL6VDEHGJ6Z4HMRU6HM6XIDUCMSWCEECCUH45FT3FQVLUDOEZGQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.510141709 (87h26m45.381224754s) [noise=true , address=145.79.227.38:4977 , piece_id=JZVULFGKR65GE24SFP5U7LHINOZIZ2BXIGQSBTZUSPXEWXEQA32A, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.510428084 (87h26m45.381511129s) [noise=true , address=145.79.227.32:4834 , piece_id=N6VJKJMTE5W3HWGCXDGPTLB3BHSFLWBMESU57UOSWPNBM55DZ4MQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:46.567032201 (87h26m46.438115236s) [noise=true , address=77.90.31.15:4698 , piece_id=2Y3GG5Y2QUJ6PFAYODA4V63N4CQXCTLRNH7LISWCXQWOAVCBSULQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 06:55:47.200741008 (87h26m47.071824053s) [upload=76331]: Next returned nil segment Dec 9 06:55:47.200756667 (87h26m47.071839702s) [upload=76331]: Waiting for error group managing segments... Dec 9 06:55:47.488289311 (87h26m47.359372346s) [noise=true , address=145.79.227.41:4086 , piece_id=GXH4GU22OG3BZQSHTK25AHWPFCYKCQEN72NXEPYXO4ZUIW2RSDEA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.488813710 (87h26m47.359896755s) [noise=true , address=145.79.227.21:4473 , piece_id=E4G6ZYPNOHCKQ6FYK4I3Y5V36C7RX5TOKVCQKN2VKHFWV5J62VEA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.488891818 (87h26m47.359974853s) [noise=true , address=145.79.227.38:4977 , piece_id=JZVULFGKR65GE24SFP5U7LHINOZIZ2BXIGQSBTZUSPXEWXEQA32A, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.489842157 (87h26m47.360925202s) [noise=true , address=145.79.227.24:4582 , piece_id=EAMAU5U3OBF5RA762ZYOARJZ4J2J4MXNEKWKY3JWGT3BX76OB6XQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.489996712 (87h26m47.361079747s) [noise=true , address=145.79.227.32:4834 , piece_id=N6VJKJMTE5W3HWGCXDGPTLB3BHSFLWBMESU57UOSWPNBM55DZ4MQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.495771257 (87h26m47.366854302s) [noise=true , address=145.79.227.34:4905 , piece_id=RUHC4SR4EO2IF2KEDJNYZOY33EAKF4JR4FTM62NZJCUD2XPO7N6Q, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.496288763 (87h26m47.367371808s) [noise=true , address=145.79.227.25:4618 , piece_id=K2Y25ECEZ6SEYOV5NPG2EEWFVN5IFFVM54QPK4ZGWZQRNE3I3BCA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.500744707 (87h26m47.371827752s) [noise=true , address=145.79.227.31:4797 , piece_id=AW3Q7GXNWAU5T5Z562KBKXYZPJEL5IRR6XB7EZZPZXUJMBQNOSBA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.501416546 (87h26m47.372499591s) [noise=true , address=145.79.227.16:4909 , piece_id=7TX6RV44XTP4C5DPVEPTSV6ECNGUSXU6G33T6L3AUCFNQLISNHOQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.501422417 (87h26m47.372505462s) [noise=true , address=145.79.227.23:4547 , piece_id=UDL6VDEHGJ6Z4HMRU6HM6XIDUCMSWCEECCUH45FT3FQVLUDOEZGQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.514960194 (87h26m47.386043249s) [noise=true , address=145.79.227.24:4581 , piece_id=WNL2FDFC4N4U5TX522OSVZDEBNO6VVZTL4TOS2C65KHY2CAMHQHA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516209051 (87h26m47.387292086s) [noise=true , address=145.79.227.29:4941 , piece_id=UDD375EXXARYRWOXMWZF25HB7VW4YVWQEFESPW4JNNB4MG7QPPSA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516598713 (87h26m47.387681758s) [noise=true , address=145.79.227.29:4942 , piece_id=SMZGQG7OUYGCHPPJLIS3PH5Y3VU65I46D4NYL2PXCBRSPQRSPZEQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516607911 (87h26m47.387690946s) [seg_pos={0 0}, upload=76331]: Segment reached optimal threshold of 13 pieces. Dec 9 06:55:47.516785880 (87h26m47.387868915s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u): protocol: expected piece hash; context canceled; EOF Dec 9 06:55:47.516824383 (87h26m47.387907448s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Long tail cancellation due to optimal threshold Dec 9 06:55:47.517129214 (87h26m47.388212259s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 06:55:47.517138261 (87h26m47.388221306s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Long tail cancellation due to optimal threshold Dec 9 06:55:47.517375302 (87h26m47.388458337s) [seg_pos={0 0}, upload=76331]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 06:55:47.517466135 (87h26m47.38854917s) [upload=76331]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 06:55:47.517473488 (87h26m47.388556534s) [upload=76331]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:00:17.830318594 (87h31m17.701401639s) [upload=76331]: Done uploading segments... Dec 9 07:00:17.830402994 (87h31m17.70148605s) [upload=76331]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.672914353s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [......X................X...X...X............................] duration: 5.891316071s trace: https://jaeger.dev.storj.io/trace/5cb4843126cfbf92 FAIL (2025-12-09 07:16:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.891537631s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.633083024s trace: https://jaeger.dev.storj.io/trace/3a1fb019f7c2965d FAIL (2025-12-09 07:15:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 06:55:45.259556946 (87h26m45.130639981s) [upload=76331]: Starting upload Dec 9 06:55:45.259623222 (87h26m45.130706257s) [upload=76331]: Done starting upload Dec 9 06:55:45.259658870 (87h26m45.130741905s) [upload=76331]: Uploading segments... Dec 9 06:55:45.259795500 (87h26m45.130878535s) [upload=76331]: Got next segment. Inline: false Dec 9 06:55:45.259821169 (87h26m45.130904204s) [upload=76331]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 06:55:45.259825137 (87h26m45.130908172s) [upload=76331]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 06:55:45.506331314 (87h26m45.377414359s) [seg_pos={0 0}, upload=76331]: Begin upload segment... Dec 9 06:55:45.506397410 (87h26m45.377480445s) [seg_pos={0 0}, upload=76331]: Done begin upload segment. Dec 9 06:55:45.506452925 (87h26m45.37753596s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.506473765 (87h26m45.3775568s) [noise=true , address=77.90.31.15:4698 , piece_id=2Y3GG5Y2QUJ6PFAYODA4V63N4CQXCTLRNH7LISWCXQWOAVCBSULQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507055783 (87h26m45.378138818s) [noise=true , address=145.79.227.41:4086 , piece_id=GXH4GU22OG3BZQSHTK25AHWPFCYKCQEN72NXEPYXO4ZUIW2RSDEA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507493356 (87h26m45.378576391s) [noise=true , address=145.79.227.29:4942 , piece_id=SMZGQG7OUYGCHPPJLIS3PH5Y3VU65I46D4NYL2PXCBRSPQRSPZEQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507543853 (87h26m45.378626898s) [noise=true , address=145.79.227.16:4909 , piece_id=7TX6RV44XTP4C5DPVEPTSV6ECNGUSXU6G33T6L3AUCFNQLISNHOQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507799740 (87h26m45.378882775s) [noise=true , address=145.79.227.29:4941 , piece_id=UDD375EXXARYRWOXMWZF25HB7VW4YVWQEFESPW4JNNB4MG7QPPSA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507920349 (87h26m45.379003384s) [noise=true , address=145.79.227.31:4797 , piece_id=AW3Q7GXNWAU5T5Z562KBKXYZPJEL5IRR6XB7EZZPZXUJMBQNOSBA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508339767 (87h26m45.379422802s) [noise=true , address=145.79.227.34:4905 , piece_id=RUHC4SR4EO2IF2KEDJNYZOY33EAKF4JR4FTM62NZJCUD2XPO7N6Q, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508566930 (87h26m45.379649965s) [noise=true , address=145.79.227.25:4618 , piece_id=K2Y25ECEZ6SEYOV5NPG2EEWFVN5IFFVM54QPK4ZGWZQRNE3I3BCA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508739269 (87h26m45.379822314s) [noise=true , address=145.79.227.24:4581 , piece_id=WNL2FDFC4N4U5TX522OSVZDEBNO6VVZTL4TOS2C65KHY2CAMHQHA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509134110 (87h26m45.380217145s) [noise=true , address=145.79.227.24:4582 , piece_id=EAMAU5U3OBF5RA762ZYOARJZ4J2J4MXNEKWKY3JWGT3BX76OB6XQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509258507 (87h26m45.380341543s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509546506 (87h26m45.380629541s) [noise=true , address=145.79.227.21:4473 , piece_id=E4G6ZYPNOHCKQ6FYK4I3Y5V36C7RX5TOKVCQKN2VKHFWV5J62VEA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509843851 (87h26m45.380926896s) [noise=true , address=145.79.227.23:4547 , piece_id=UDL6VDEHGJ6Z4HMRU6HM6XIDUCMSWCEECCUH45FT3FQVLUDOEZGQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.510141709 (87h26m45.381224754s) [noise=true , address=145.79.227.38:4977 , piece_id=JZVULFGKR65GE24SFP5U7LHINOZIZ2BXIGQSBTZUSPXEWXEQA32A, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.510428084 (87h26m45.381511129s) [noise=true , address=145.79.227.32:4834 , piece_id=N6VJKJMTE5W3HWGCXDGPTLB3BHSFLWBMESU57UOSWPNBM55DZ4MQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:46.567032201 (87h26m46.438115236s) [noise=true , address=77.90.31.15:4698 , piece_id=2Y3GG5Y2QUJ6PFAYODA4V63N4CQXCTLRNH7LISWCXQWOAVCBSULQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 06:55:47.200741008 (87h26m47.071824053s) [upload=76331]: Next returned nil segment Dec 9 06:55:47.200756667 (87h26m47.071839702s) [upload=76331]: Waiting for error group managing segments... Dec 9 06:55:47.488289311 (87h26m47.359372346s) [noise=true , address=145.79.227.41:4086 , piece_id=GXH4GU22OG3BZQSHTK25AHWPFCYKCQEN72NXEPYXO4ZUIW2RSDEA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.488813710 (87h26m47.359896755s) [noise=true , address=145.79.227.21:4473 , piece_id=E4G6ZYPNOHCKQ6FYK4I3Y5V36C7RX5TOKVCQKN2VKHFWV5J62VEA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.488891818 (87h26m47.359974853s) [noise=true , address=145.79.227.38:4977 , piece_id=JZVULFGKR65GE24SFP5U7LHINOZIZ2BXIGQSBTZUSPXEWXEQA32A, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.489842157 (87h26m47.360925202s) [noise=true , address=145.79.227.24:4582 , piece_id=EAMAU5U3OBF5RA762ZYOARJZ4J2J4MXNEKWKY3JWGT3BX76OB6XQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.489996712 (87h26m47.361079747s) [noise=true , address=145.79.227.32:4834 , piece_id=N6VJKJMTE5W3HWGCXDGPTLB3BHSFLWBMESU57UOSWPNBM55DZ4MQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.495771257 (87h26m47.366854302s) [noise=true , address=145.79.227.34:4905 , piece_id=RUHC4SR4EO2IF2KEDJNYZOY33EAKF4JR4FTM62NZJCUD2XPO7N6Q, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.496288763 (87h26m47.367371808s) [noise=true , address=145.79.227.25:4618 , piece_id=K2Y25ECEZ6SEYOV5NPG2EEWFVN5IFFVM54QPK4ZGWZQRNE3I3BCA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.500744707 (87h26m47.371827752s) [noise=true , address=145.79.227.31:4797 , piece_id=AW3Q7GXNWAU5T5Z562KBKXYZPJEL5IRR6XB7EZZPZXUJMBQNOSBA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.501416546 (87h26m47.372499591s) [noise=true , address=145.79.227.16:4909 , piece_id=7TX6RV44XTP4C5DPVEPTSV6ECNGUSXU6G33T6L3AUCFNQLISNHOQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.501422417 (87h26m47.372505462s) [noise=true , address=145.79.227.23:4547 , piece_id=UDL6VDEHGJ6Z4HMRU6HM6XIDUCMSWCEECCUH45FT3FQVLUDOEZGQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.514960194 (87h26m47.386043249s) [noise=true , address=145.79.227.24:4581 , piece_id=WNL2FDFC4N4U5TX522OSVZDEBNO6VVZTL4TOS2C65KHY2CAMHQHA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516209051 (87h26m47.387292086s) [noise=true , address=145.79.227.29:4941 , piece_id=UDD375EXXARYRWOXMWZF25HB7VW4YVWQEFESPW4JNNB4MG7QPPSA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516598713 (87h26m47.387681758s) [noise=true , address=145.79.227.29:4942 , piece_id=SMZGQG7OUYGCHPPJLIS3PH5Y3VU65I46D4NYL2PXCBRSPQRSPZEQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516607911 (87h26m47.387690946s) [seg_pos={0 0}, upload=76331]: Segment reached optimal threshold of 13 pieces. Dec 9 06:55:47.516785880 (87h26m47.387868915s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u): protocol: expected piece hash; context canceled; EOF Dec 9 06:55:47.516824383 (87h26m47.387907448s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Long tail cancellation due to optimal threshold Dec 9 06:55:47.517129214 (87h26m47.388212259s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 06:55:47.517138261 (87h26m47.388221306s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Long tail cancellation due to optimal threshold Dec 9 06:55:47.517375302 (87h26m47.388458337s) [seg_pos={0 0}, upload=76331]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 06:55:47.517466135 (87h26m47.38854917s) [upload=76331]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 06:55:47.517473488 (87h26m47.388556534s) [upload=76331]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:00:17.830318594 (87h31m17.701401639s) [upload=76331]: Done uploading segments... Dec 9 07:00:17.830402994 (87h31m17.70148605s) [upload=76331]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.633110777s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 4.897318494s trace: https://jaeger.dev.storj.io/trace/488f36bcf1592ab1 FAIL (2025-12-09 07:15:42 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.897340146s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.724878409s trace: https://jaeger.dev.storj.io/trace/3a1fb019f7c2965d FAIL (2025-12-09 07:15:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 06:55:45.259556946 (87h26m45.130639981s) [upload=76331]: Starting upload Dec 9 06:55:45.259623222 (87h26m45.130706257s) [upload=76331]: Done starting upload Dec 9 06:55:45.259658870 (87h26m45.130741905s) [upload=76331]: Uploading segments... Dec 9 06:55:45.259795500 (87h26m45.130878535s) [upload=76331]: Got next segment. Inline: false Dec 9 06:55:45.259821169 (87h26m45.130904204s) [upload=76331]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 06:55:45.259825137 (87h26m45.130908172s) [upload=76331]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 06:55:45.506331314 (87h26m45.377414359s) [seg_pos={0 0}, upload=76331]: Begin upload segment... Dec 9 06:55:45.506397410 (87h26m45.377480445s) [seg_pos={0 0}, upload=76331]: Done begin upload segment. Dec 9 06:55:45.506452925 (87h26m45.37753596s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.506473765 (87h26m45.3775568s) [noise=true , address=77.90.31.15:4698 , piece_id=2Y3GG5Y2QUJ6PFAYODA4V63N4CQXCTLRNH7LISWCXQWOAVCBSULQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507055783 (87h26m45.378138818s) [noise=true , address=145.79.227.41:4086 , piece_id=GXH4GU22OG3BZQSHTK25AHWPFCYKCQEN72NXEPYXO4ZUIW2RSDEA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507493356 (87h26m45.378576391s) [noise=true , address=145.79.227.29:4942 , piece_id=SMZGQG7OUYGCHPPJLIS3PH5Y3VU65I46D4NYL2PXCBRSPQRSPZEQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507543853 (87h26m45.378626898s) [noise=true , address=145.79.227.16:4909 , piece_id=7TX6RV44XTP4C5DPVEPTSV6ECNGUSXU6G33T6L3AUCFNQLISNHOQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507799740 (87h26m45.378882775s) [noise=true , address=145.79.227.29:4941 , piece_id=UDD375EXXARYRWOXMWZF25HB7VW4YVWQEFESPW4JNNB4MG7QPPSA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507920349 (87h26m45.379003384s) [noise=true , address=145.79.227.31:4797 , piece_id=AW3Q7GXNWAU5T5Z562KBKXYZPJEL5IRR6XB7EZZPZXUJMBQNOSBA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508339767 (87h26m45.379422802s) [noise=true , address=145.79.227.34:4905 , piece_id=RUHC4SR4EO2IF2KEDJNYZOY33EAKF4JR4FTM62NZJCUD2XPO7N6Q, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508566930 (87h26m45.379649965s) [noise=true , address=145.79.227.25:4618 , piece_id=K2Y25ECEZ6SEYOV5NPG2EEWFVN5IFFVM54QPK4ZGWZQRNE3I3BCA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508739269 (87h26m45.379822314s) [noise=true , address=145.79.227.24:4581 , piece_id=WNL2FDFC4N4U5TX522OSVZDEBNO6VVZTL4TOS2C65KHY2CAMHQHA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509134110 (87h26m45.380217145s) [noise=true , address=145.79.227.24:4582 , piece_id=EAMAU5U3OBF5RA762ZYOARJZ4J2J4MXNEKWKY3JWGT3BX76OB6XQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509258507 (87h26m45.380341543s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509546506 (87h26m45.380629541s) [noise=true , address=145.79.227.21:4473 , piece_id=E4G6ZYPNOHCKQ6FYK4I3Y5V36C7RX5TOKVCQKN2VKHFWV5J62VEA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509843851 (87h26m45.380926896s) [noise=true , address=145.79.227.23:4547 , piece_id=UDL6VDEHGJ6Z4HMRU6HM6XIDUCMSWCEECCUH45FT3FQVLUDOEZGQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.510141709 (87h26m45.381224754s) [noise=true , address=145.79.227.38:4977 , piece_id=JZVULFGKR65GE24SFP5U7LHINOZIZ2BXIGQSBTZUSPXEWXEQA32A, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.510428084 (87h26m45.381511129s) [noise=true , address=145.79.227.32:4834 , piece_id=N6VJKJMTE5W3HWGCXDGPTLB3BHSFLWBMESU57UOSWPNBM55DZ4MQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:46.567032201 (87h26m46.438115236s) [noise=true , address=77.90.31.15:4698 , piece_id=2Y3GG5Y2QUJ6PFAYODA4V63N4CQXCTLRNH7LISWCXQWOAVCBSULQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 06:55:47.200741008 (87h26m47.071824053s) [upload=76331]: Next returned nil segment Dec 9 06:55:47.200756667 (87h26m47.071839702s) [upload=76331]: Waiting for error group managing segments... Dec 9 06:55:47.488289311 (87h26m47.359372346s) [noise=true , address=145.79.227.41:4086 , piece_id=GXH4GU22OG3BZQSHTK25AHWPFCYKCQEN72NXEPYXO4ZUIW2RSDEA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.488813710 (87h26m47.359896755s) [noise=true , address=145.79.227.21:4473 , piece_id=E4G6ZYPNOHCKQ6FYK4I3Y5V36C7RX5TOKVCQKN2VKHFWV5J62VEA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.488891818 (87h26m47.359974853s) [noise=true , address=145.79.227.38:4977 , piece_id=JZVULFGKR65GE24SFP5U7LHINOZIZ2BXIGQSBTZUSPXEWXEQA32A, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.489842157 (87h26m47.360925202s) [noise=true , address=145.79.227.24:4582 , piece_id=EAMAU5U3OBF5RA762ZYOARJZ4J2J4MXNEKWKY3JWGT3BX76OB6XQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.489996712 (87h26m47.361079747s) [noise=true , address=145.79.227.32:4834 , piece_id=N6VJKJMTE5W3HWGCXDGPTLB3BHSFLWBMESU57UOSWPNBM55DZ4MQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.495771257 (87h26m47.366854302s) [noise=true , address=145.79.227.34:4905 , piece_id=RUHC4SR4EO2IF2KEDJNYZOY33EAKF4JR4FTM62NZJCUD2XPO7N6Q, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.496288763 (87h26m47.367371808s) [noise=true , address=145.79.227.25:4618 , piece_id=K2Y25ECEZ6SEYOV5NPG2EEWFVN5IFFVM54QPK4ZGWZQRNE3I3BCA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.500744707 (87h26m47.371827752s) [noise=true , address=145.79.227.31:4797 , piece_id=AW3Q7GXNWAU5T5Z562KBKXYZPJEL5IRR6XB7EZZPZXUJMBQNOSBA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.501416546 (87h26m47.372499591s) [noise=true , address=145.79.227.16:4909 , piece_id=7TX6RV44XTP4C5DPVEPTSV6ECNGUSXU6G33T6L3AUCFNQLISNHOQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.501422417 (87h26m47.372505462s) [noise=true , address=145.79.227.23:4547 , piece_id=UDL6VDEHGJ6Z4HMRU6HM6XIDUCMSWCEECCUH45FT3FQVLUDOEZGQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.514960194 (87h26m47.386043249s) [noise=true , address=145.79.227.24:4581 , piece_id=WNL2FDFC4N4U5TX522OSVZDEBNO6VVZTL4TOS2C65KHY2CAMHQHA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516209051 (87h26m47.387292086s) [noise=true , address=145.79.227.29:4941 , piece_id=UDD375EXXARYRWOXMWZF25HB7VW4YVWQEFESPW4JNNB4MG7QPPSA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516598713 (87h26m47.387681758s) [noise=true , address=145.79.227.29:4942 , piece_id=SMZGQG7OUYGCHPPJLIS3PH5Y3VU65I46D4NYL2PXCBRSPQRSPZEQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516607911 (87h26m47.387690946s) [seg_pos={0 0}, upload=76331]: Segment reached optimal threshold of 13 pieces. Dec 9 06:55:47.516785880 (87h26m47.387868915s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u): protocol: expected piece hash; context canceled; EOF Dec 9 06:55:47.516824383 (87h26m47.387907448s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Long tail cancellation due to optimal threshold Dec 9 06:55:47.517129214 (87h26m47.388212259s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 06:55:47.517138261 (87h26m47.388221306s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Long tail cancellation due to optimal threshold Dec 9 06:55:47.517375302 (87h26m47.388458337s) [seg_pos={0 0}, upload=76331]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 06:55:47.517466135 (87h26m47.38854917s) [upload=76331]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 06:55:47.517473488 (87h26m47.388556534s) [upload=76331]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:00:17.830318594 (87h31m17.701401639s) [upload=76331]: Done uploading segments... Dec 9 07:00:17.830402994 (87h31m17.70148605s) [upload=76331]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.724904068s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..X...............X....X....................................] duration: 3.644425509s trace: https://jaeger.dev.storj.io/trace/2c78734d9bae1209 FAIL (2025-12-09 07:15:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.64444708s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......X................X...X...X...........................] duration: 5.931184628s trace: https://jaeger.dev.storj.io/trace/5cb4843126cfbf92 FAIL (2025-12-09 07:15:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.93120669s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 4.997037859s trace: https://jaeger.dev.storj.io/trace/488f36bcf1592ab1 FAIL (2025-12-09 07:14:59 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.997066423s core uplink eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.......................................X...................] duration: 5.877301557s trace: https://jaeger.dev.storj.io/trace/3a1fb019f7c2965d FAIL (2025-12-09 07:14:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 06:55:45.259556946 (87h26m45.130639981s) [upload=76331]: Starting upload Dec 9 06:55:45.259623222 (87h26m45.130706257s) [upload=76331]: Done starting upload Dec 9 06:55:45.259658870 (87h26m45.130741905s) [upload=76331]: Uploading segments... Dec 9 06:55:45.259795500 (87h26m45.130878535s) [upload=76331]: Got next segment. Inline: false Dec 9 06:55:45.259821169 (87h26m45.130904204s) [upload=76331]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 06:55:45.259825137 (87h26m45.130908172s) [upload=76331]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 06:55:45.506331314 (87h26m45.377414359s) [seg_pos={0 0}, upload=76331]: Begin upload segment... Dec 9 06:55:45.506397410 (87h26m45.377480445s) [seg_pos={0 0}, upload=76331]: Done begin upload segment. Dec 9 06:55:45.506452925 (87h26m45.37753596s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.506473765 (87h26m45.3775568s) [noise=true , address=77.90.31.15:4698 , piece_id=2Y3GG5Y2QUJ6PFAYODA4V63N4CQXCTLRNH7LISWCXQWOAVCBSULQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507055783 (87h26m45.378138818s) [noise=true , address=145.79.227.41:4086 , piece_id=GXH4GU22OG3BZQSHTK25AHWPFCYKCQEN72NXEPYXO4ZUIW2RSDEA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507493356 (87h26m45.378576391s) [noise=true , address=145.79.227.29:4942 , piece_id=SMZGQG7OUYGCHPPJLIS3PH5Y3VU65I46D4NYL2PXCBRSPQRSPZEQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507543853 (87h26m45.378626898s) [noise=true , address=145.79.227.16:4909 , piece_id=7TX6RV44XTP4C5DPVEPTSV6ECNGUSXU6G33T6L3AUCFNQLISNHOQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507799740 (87h26m45.378882775s) [noise=true , address=145.79.227.29:4941 , piece_id=UDD375EXXARYRWOXMWZF25HB7VW4YVWQEFESPW4JNNB4MG7QPPSA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507920349 (87h26m45.379003384s) [noise=true , address=145.79.227.31:4797 , piece_id=AW3Q7GXNWAU5T5Z562KBKXYZPJEL5IRR6XB7EZZPZXUJMBQNOSBA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508339767 (87h26m45.379422802s) [noise=true , address=145.79.227.34:4905 , piece_id=RUHC4SR4EO2IF2KEDJNYZOY33EAKF4JR4FTM62NZJCUD2XPO7N6Q, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508566930 (87h26m45.379649965s) [noise=true , address=145.79.227.25:4618 , piece_id=K2Y25ECEZ6SEYOV5NPG2EEWFVN5IFFVM54QPK4ZGWZQRNE3I3BCA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508739269 (87h26m45.379822314s) [noise=true , address=145.79.227.24:4581 , piece_id=WNL2FDFC4N4U5TX522OSVZDEBNO6VVZTL4TOS2C65KHY2CAMHQHA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509134110 (87h26m45.380217145s) [noise=true , address=145.79.227.24:4582 , piece_id=EAMAU5U3OBF5RA762ZYOARJZ4J2J4MXNEKWKY3JWGT3BX76OB6XQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509258507 (87h26m45.380341543s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509546506 (87h26m45.380629541s) [noise=true , address=145.79.227.21:4473 , piece_id=E4G6ZYPNOHCKQ6FYK4I3Y5V36C7RX5TOKVCQKN2VKHFWV5J62VEA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509843851 (87h26m45.380926896s) [noise=true , address=145.79.227.23:4547 , piece_id=UDL6VDEHGJ6Z4HMRU6HM6XIDUCMSWCEECCUH45FT3FQVLUDOEZGQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.510141709 (87h26m45.381224754s) [noise=true , address=145.79.227.38:4977 , piece_id=JZVULFGKR65GE24SFP5U7LHINOZIZ2BXIGQSBTZUSPXEWXEQA32A, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.510428084 (87h26m45.381511129s) [noise=true , address=145.79.227.32:4834 , piece_id=N6VJKJMTE5W3HWGCXDGPTLB3BHSFLWBMESU57UOSWPNBM55DZ4MQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:46.567032201 (87h26m46.438115236s) [noise=true , address=77.90.31.15:4698 , piece_id=2Y3GG5Y2QUJ6PFAYODA4V63N4CQXCTLRNH7LISWCXQWOAVCBSULQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 06:55:47.200741008 (87h26m47.071824053s) [upload=76331]: Next returned nil segment Dec 9 06:55:47.200756667 (87h26m47.071839702s) [upload=76331]: Waiting for error group managing segments... Dec 9 06:55:47.488289311 (87h26m47.359372346s) [noise=true , address=145.79.227.41:4086 , piece_id=GXH4GU22OG3BZQSHTK25AHWPFCYKCQEN72NXEPYXO4ZUIW2RSDEA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.488813710 (87h26m47.359896755s) [noise=true , address=145.79.227.21:4473 , piece_id=E4G6ZYPNOHCKQ6FYK4I3Y5V36C7RX5TOKVCQKN2VKHFWV5J62VEA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.488891818 (87h26m47.359974853s) [noise=true , address=145.79.227.38:4977 , piece_id=JZVULFGKR65GE24SFP5U7LHINOZIZ2BXIGQSBTZUSPXEWXEQA32A, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.489842157 (87h26m47.360925202s) [noise=true , address=145.79.227.24:4582 , piece_id=EAMAU5U3OBF5RA762ZYOARJZ4J2J4MXNEKWKY3JWGT3BX76OB6XQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.489996712 (87h26m47.361079747s) [noise=true , address=145.79.227.32:4834 , piece_id=N6VJKJMTE5W3HWGCXDGPTLB3BHSFLWBMESU57UOSWPNBM55DZ4MQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.495771257 (87h26m47.366854302s) [noise=true , address=145.79.227.34:4905 , piece_id=RUHC4SR4EO2IF2KEDJNYZOY33EAKF4JR4FTM62NZJCUD2XPO7N6Q, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.496288763 (87h26m47.367371808s) [noise=true , address=145.79.227.25:4618 , piece_id=K2Y25ECEZ6SEYOV5NPG2EEWFVN5IFFVM54QPK4ZGWZQRNE3I3BCA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.500744707 (87h26m47.371827752s) [noise=true , address=145.79.227.31:4797 , piece_id=AW3Q7GXNWAU5T5Z562KBKXYZPJEL5IRR6XB7EZZPZXUJMBQNOSBA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.501416546 (87h26m47.372499591s) [noise=true , address=145.79.227.16:4909 , piece_id=7TX6RV44XTP4C5DPVEPTSV6ECNGUSXU6G33T6L3AUCFNQLISNHOQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.501422417 (87h26m47.372505462s) [noise=true , address=145.79.227.23:4547 , piece_id=UDL6VDEHGJ6Z4HMRU6HM6XIDUCMSWCEECCUH45FT3FQVLUDOEZGQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.514960194 (87h26m47.386043249s) [noise=true , address=145.79.227.24:4581 , piece_id=WNL2FDFC4N4U5TX522OSVZDEBNO6VVZTL4TOS2C65KHY2CAMHQHA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516209051 (87h26m47.387292086s) [noise=true , address=145.79.227.29:4941 , piece_id=UDD375EXXARYRWOXMWZF25HB7VW4YVWQEFESPW4JNNB4MG7QPPSA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516598713 (87h26m47.387681758s) [noise=true , address=145.79.227.29:4942 , piece_id=SMZGQG7OUYGCHPPJLIS3PH5Y3VU65I46D4NYL2PXCBRSPQRSPZEQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516607911 (87h26m47.387690946s) [seg_pos={0 0}, upload=76331]: Segment reached optimal threshold of 13 pieces. Dec 9 06:55:47.516785880 (87h26m47.387868915s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u): protocol: expected piece hash; context canceled; EOF Dec 9 06:55:47.516824383 (87h26m47.387907448s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Long tail cancellation due to optimal threshold Dec 9 06:55:47.517129214 (87h26m47.388212259s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 06:55:47.517138261 (87h26m47.388221306s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Long tail cancellation due to optimal threshold Dec 9 06:55:47.517375302 (87h26m47.388458337s) [seg_pos={0 0}, upload=76331]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 06:55:47.517466135 (87h26m47.38854917s) [upload=76331]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 06:55:47.517473488 (87h26m47.388556534s) [upload=76331]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:00:17.830318594 (87h31m17.701401639s) [upload=76331]: Done uploading segments... Dec 9 07:00:17.830402994 (87h31m17.70148605s) [upload=76331]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.877334809s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........X................X...X...X..........................] duration: 5.671870722s trace: https://jaeger.dev.storj.io/trace/5cb4843126cfbf92 FAIL (2025-12-09 07:14:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.671936008s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...X...............X....X...................................] duration: 3.470939509s trace: https://jaeger.dev.storj.io/trace/2c78734d9bae1209 FAIL (2025-12-09 07:14:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.470978303s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.......................................X..................] duration: 5.910837407s trace: https://jaeger.dev.storj.io/trace/3a1fb019f7c2965d FAIL (2025-12-09 07:14:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 06:55:45.259556946 (87h26m45.130639981s) [upload=76331]: Starting upload Dec 9 06:55:45.259623222 (87h26m45.130706257s) [upload=76331]: Done starting upload Dec 9 06:55:45.259658870 (87h26m45.130741905s) [upload=76331]: Uploading segments... Dec 9 06:55:45.259795500 (87h26m45.130878535s) [upload=76331]: Got next segment. Inline: false Dec 9 06:55:45.259821169 (87h26m45.130904204s) [upload=76331]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 06:55:45.259825137 (87h26m45.130908172s) [upload=76331]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 06:55:45.506331314 (87h26m45.377414359s) [seg_pos={0 0}, upload=76331]: Begin upload segment... Dec 9 06:55:45.506397410 (87h26m45.377480445s) [seg_pos={0 0}, upload=76331]: Done begin upload segment. Dec 9 06:55:45.506452925 (87h26m45.37753596s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.506473765 (87h26m45.3775568s) [noise=true , address=77.90.31.15:4698 , piece_id=2Y3GG5Y2QUJ6PFAYODA4V63N4CQXCTLRNH7LISWCXQWOAVCBSULQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507055783 (87h26m45.378138818s) [noise=true , address=145.79.227.41:4086 , piece_id=GXH4GU22OG3BZQSHTK25AHWPFCYKCQEN72NXEPYXO4ZUIW2RSDEA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507493356 (87h26m45.378576391s) [noise=true , address=145.79.227.29:4942 , piece_id=SMZGQG7OUYGCHPPJLIS3PH5Y3VU65I46D4NYL2PXCBRSPQRSPZEQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507543853 (87h26m45.378626898s) [noise=true , address=145.79.227.16:4909 , piece_id=7TX6RV44XTP4C5DPVEPTSV6ECNGUSXU6G33T6L3AUCFNQLISNHOQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507799740 (87h26m45.378882775s) [noise=true , address=145.79.227.29:4941 , piece_id=UDD375EXXARYRWOXMWZF25HB7VW4YVWQEFESPW4JNNB4MG7QPPSA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507920349 (87h26m45.379003384s) [noise=true , address=145.79.227.31:4797 , piece_id=AW3Q7GXNWAU5T5Z562KBKXYZPJEL5IRR6XB7EZZPZXUJMBQNOSBA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508339767 (87h26m45.379422802s) [noise=true , address=145.79.227.34:4905 , piece_id=RUHC4SR4EO2IF2KEDJNYZOY33EAKF4JR4FTM62NZJCUD2XPO7N6Q, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508566930 (87h26m45.379649965s) [noise=true , address=145.79.227.25:4618 , piece_id=K2Y25ECEZ6SEYOV5NPG2EEWFVN5IFFVM54QPK4ZGWZQRNE3I3BCA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508739269 (87h26m45.379822314s) [noise=true , address=145.79.227.24:4581 , piece_id=WNL2FDFC4N4U5TX522OSVZDEBNO6VVZTL4TOS2C65KHY2CAMHQHA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509134110 (87h26m45.380217145s) [noise=true , address=145.79.227.24:4582 , piece_id=EAMAU5U3OBF5RA762ZYOARJZ4J2J4MXNEKWKY3JWGT3BX76OB6XQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509258507 (87h26m45.380341543s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509546506 (87h26m45.380629541s) [noise=true , address=145.79.227.21:4473 , piece_id=E4G6ZYPNOHCKQ6FYK4I3Y5V36C7RX5TOKVCQKN2VKHFWV5J62VEA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509843851 (87h26m45.380926896s) [noise=true , address=145.79.227.23:4547 , piece_id=UDL6VDEHGJ6Z4HMRU6HM6XIDUCMSWCEECCUH45FT3FQVLUDOEZGQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.510141709 (87h26m45.381224754s) [noise=true , address=145.79.227.38:4977 , piece_id=JZVULFGKR65GE24SFP5U7LHINOZIZ2BXIGQSBTZUSPXEWXEQA32A, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.510428084 (87h26m45.381511129s) [noise=true , address=145.79.227.32:4834 , piece_id=N6VJKJMTE5W3HWGCXDGPTLB3BHSFLWBMESU57UOSWPNBM55DZ4MQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:46.567032201 (87h26m46.438115236s) [noise=true , address=77.90.31.15:4698 , piece_id=2Y3GG5Y2QUJ6PFAYODA4V63N4CQXCTLRNH7LISWCXQWOAVCBSULQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 06:55:47.200741008 (87h26m47.071824053s) [upload=76331]: Next returned nil segment Dec 9 06:55:47.200756667 (87h26m47.071839702s) [upload=76331]: Waiting for error group managing segments... Dec 9 06:55:47.488289311 (87h26m47.359372346s) [noise=true , address=145.79.227.41:4086 , piece_id=GXH4GU22OG3BZQSHTK25AHWPFCYKCQEN72NXEPYXO4ZUIW2RSDEA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.488813710 (87h26m47.359896755s) [noise=true , address=145.79.227.21:4473 , piece_id=E4G6ZYPNOHCKQ6FYK4I3Y5V36C7RX5TOKVCQKN2VKHFWV5J62VEA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.488891818 (87h26m47.359974853s) [noise=true , address=145.79.227.38:4977 , piece_id=JZVULFGKR65GE24SFP5U7LHINOZIZ2BXIGQSBTZUSPXEWXEQA32A, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.489842157 (87h26m47.360925202s) [noise=true , address=145.79.227.24:4582 , piece_id=EAMAU5U3OBF5RA762ZYOARJZ4J2J4MXNEKWKY3JWGT3BX76OB6XQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.489996712 (87h26m47.361079747s) [noise=true , address=145.79.227.32:4834 , piece_id=N6VJKJMTE5W3HWGCXDGPTLB3BHSFLWBMESU57UOSWPNBM55DZ4MQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.495771257 (87h26m47.366854302s) [noise=true , address=145.79.227.34:4905 , piece_id=RUHC4SR4EO2IF2KEDJNYZOY33EAKF4JR4FTM62NZJCUD2XPO7N6Q, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.496288763 (87h26m47.367371808s) [noise=true , address=145.79.227.25:4618 , piece_id=K2Y25ECEZ6SEYOV5NPG2EEWFVN5IFFVM54QPK4ZGWZQRNE3I3BCA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.500744707 (87h26m47.371827752s) [noise=true , address=145.79.227.31:4797 , piece_id=AW3Q7GXNWAU5T5Z562KBKXYZPJEL5IRR6XB7EZZPZXUJMBQNOSBA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.501416546 (87h26m47.372499591s) [noise=true , address=145.79.227.16:4909 , piece_id=7TX6RV44XTP4C5DPVEPTSV6ECNGUSXU6G33T6L3AUCFNQLISNHOQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.501422417 (87h26m47.372505462s) [noise=true , address=145.79.227.23:4547 , piece_id=UDL6VDEHGJ6Z4HMRU6HM6XIDUCMSWCEECCUH45FT3FQVLUDOEZGQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.514960194 (87h26m47.386043249s) [noise=true , address=145.79.227.24:4581 , piece_id=WNL2FDFC4N4U5TX522OSVZDEBNO6VVZTL4TOS2C65KHY2CAMHQHA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516209051 (87h26m47.387292086s) [noise=true , address=145.79.227.29:4941 , piece_id=UDD375EXXARYRWOXMWZF25HB7VW4YVWQEFESPW4JNNB4MG7QPPSA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516598713 (87h26m47.387681758s) [noise=true , address=145.79.227.29:4942 , piece_id=SMZGQG7OUYGCHPPJLIS3PH5Y3VU65I46D4NYL2PXCBRSPQRSPZEQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516607911 (87h26m47.387690946s) [seg_pos={0 0}, upload=76331]: Segment reached optimal threshold of 13 pieces. Dec 9 06:55:47.516785880 (87h26m47.387868915s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u): protocol: expected piece hash; context canceled; EOF Dec 9 06:55:47.516824383 (87h26m47.387907448s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Long tail cancellation due to optimal threshold Dec 9 06:55:47.517129214 (87h26m47.388212259s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 06:55:47.517138261 (87h26m47.388221306s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Long tail cancellation due to optimal threshold Dec 9 06:55:47.517375302 (87h26m47.388458337s) [seg_pos={0 0}, upload=76331]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 06:55:47.517466135 (87h26m47.38854917s) [upload=76331]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 06:55:47.517473488 (87h26m47.388556534s) [upload=76331]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:00:17.830318594 (87h31m17.701401639s) [upload=76331]: Done uploading segments... Dec 9 07:00:17.830402994 (87h31m17.70148605s) [upload=76331]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.910871491s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 4.939478636s trace: https://jaeger.dev.storj.io/trace/488f36bcf1592ab1 FAIL (2025-12-09 07:14:02 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.939509053s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....X...............X....X..................................] duration: 3.359768218s trace: https://jaeger.dev.storj.io/trace/2c78734d9bae1209 FAIL (2025-12-09 07:13:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.35979142s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.........X................X...X...X.........................] duration: 5.87666842s trace: https://jaeger.dev.storj.io/trace/5cb4843126cfbf92 FAIL (2025-12-09 07:13:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.876697265s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.......................................X.................] duration: 5.806603802s trace: https://jaeger.dev.storj.io/trace/3a1fb019f7c2965d FAIL (2025-12-09 07:12:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 06:55:45.259556946 (87h26m45.130639981s) [upload=76331]: Starting upload Dec 9 06:55:45.259623222 (87h26m45.130706257s) [upload=76331]: Done starting upload Dec 9 06:55:45.259658870 (87h26m45.130741905s) [upload=76331]: Uploading segments... Dec 9 06:55:45.259795500 (87h26m45.130878535s) [upload=76331]: Got next segment. Inline: false Dec 9 06:55:45.259821169 (87h26m45.130904204s) [upload=76331]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 06:55:45.259825137 (87h26m45.130908172s) [upload=76331]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 06:55:45.506331314 (87h26m45.377414359s) [seg_pos={0 0}, upload=76331]: Begin upload segment... Dec 9 06:55:45.506397410 (87h26m45.377480445s) [seg_pos={0 0}, upload=76331]: Done begin upload segment. Dec 9 06:55:45.506452925 (87h26m45.37753596s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.506473765 (87h26m45.3775568s) [noise=true , address=77.90.31.15:4698 , piece_id=2Y3GG5Y2QUJ6PFAYODA4V63N4CQXCTLRNH7LISWCXQWOAVCBSULQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507055783 (87h26m45.378138818s) [noise=true , address=145.79.227.41:4086 , piece_id=GXH4GU22OG3BZQSHTK25AHWPFCYKCQEN72NXEPYXO4ZUIW2RSDEA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507493356 (87h26m45.378576391s) [noise=true , address=145.79.227.29:4942 , piece_id=SMZGQG7OUYGCHPPJLIS3PH5Y3VU65I46D4NYL2PXCBRSPQRSPZEQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507543853 (87h26m45.378626898s) [noise=true , address=145.79.227.16:4909 , piece_id=7TX6RV44XTP4C5DPVEPTSV6ECNGUSXU6G33T6L3AUCFNQLISNHOQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507799740 (87h26m45.378882775s) [noise=true , address=145.79.227.29:4941 , piece_id=UDD375EXXARYRWOXMWZF25HB7VW4YVWQEFESPW4JNNB4MG7QPPSA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507920349 (87h26m45.379003384s) [noise=true , address=145.79.227.31:4797 , piece_id=AW3Q7GXNWAU5T5Z562KBKXYZPJEL5IRR6XB7EZZPZXUJMBQNOSBA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508339767 (87h26m45.379422802s) [noise=true , address=145.79.227.34:4905 , piece_id=RUHC4SR4EO2IF2KEDJNYZOY33EAKF4JR4FTM62NZJCUD2XPO7N6Q, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508566930 (87h26m45.379649965s) [noise=true , address=145.79.227.25:4618 , piece_id=K2Y25ECEZ6SEYOV5NPG2EEWFVN5IFFVM54QPK4ZGWZQRNE3I3BCA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508739269 (87h26m45.379822314s) [noise=true , address=145.79.227.24:4581 , piece_id=WNL2FDFC4N4U5TX522OSVZDEBNO6VVZTL4TOS2C65KHY2CAMHQHA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509134110 (87h26m45.380217145s) [noise=true , address=145.79.227.24:4582 , piece_id=EAMAU5U3OBF5RA762ZYOARJZ4J2J4MXNEKWKY3JWGT3BX76OB6XQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509258507 (87h26m45.380341543s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509546506 (87h26m45.380629541s) [noise=true , address=145.79.227.21:4473 , piece_id=E4G6ZYPNOHCKQ6FYK4I3Y5V36C7RX5TOKVCQKN2VKHFWV5J62VEA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509843851 (87h26m45.380926896s) [noise=true , address=145.79.227.23:4547 , piece_id=UDL6VDEHGJ6Z4HMRU6HM6XIDUCMSWCEECCUH45FT3FQVLUDOEZGQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.510141709 (87h26m45.381224754s) [noise=true , address=145.79.227.38:4977 , piece_id=JZVULFGKR65GE24SFP5U7LHINOZIZ2BXIGQSBTZUSPXEWXEQA32A, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.510428084 (87h26m45.381511129s) [noise=true , address=145.79.227.32:4834 , piece_id=N6VJKJMTE5W3HWGCXDGPTLB3BHSFLWBMESU57UOSWPNBM55DZ4MQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:46.567032201 (87h26m46.438115236s) [noise=true , address=77.90.31.15:4698 , piece_id=2Y3GG5Y2QUJ6PFAYODA4V63N4CQXCTLRNH7LISWCXQWOAVCBSULQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 06:55:47.200741008 (87h26m47.071824053s) [upload=76331]: Next returned nil segment Dec 9 06:55:47.200756667 (87h26m47.071839702s) [upload=76331]: Waiting for error group managing segments... Dec 9 06:55:47.488289311 (87h26m47.359372346s) [noise=true , address=145.79.227.41:4086 , piece_id=GXH4GU22OG3BZQSHTK25AHWPFCYKCQEN72NXEPYXO4ZUIW2RSDEA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.488813710 (87h26m47.359896755s) [noise=true , address=145.79.227.21:4473 , piece_id=E4G6ZYPNOHCKQ6FYK4I3Y5V36C7RX5TOKVCQKN2VKHFWV5J62VEA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.488891818 (87h26m47.359974853s) [noise=true , address=145.79.227.38:4977 , piece_id=JZVULFGKR65GE24SFP5U7LHINOZIZ2BXIGQSBTZUSPXEWXEQA32A, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.489842157 (87h26m47.360925202s) [noise=true , address=145.79.227.24:4582 , piece_id=EAMAU5U3OBF5RA762ZYOARJZ4J2J4MXNEKWKY3JWGT3BX76OB6XQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.489996712 (87h26m47.361079747s) [noise=true , address=145.79.227.32:4834 , piece_id=N6VJKJMTE5W3HWGCXDGPTLB3BHSFLWBMESU57UOSWPNBM55DZ4MQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.495771257 (87h26m47.366854302s) [noise=true , address=145.79.227.34:4905 , piece_id=RUHC4SR4EO2IF2KEDJNYZOY33EAKF4JR4FTM62NZJCUD2XPO7N6Q, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.496288763 (87h26m47.367371808s) [noise=true , address=145.79.227.25:4618 , piece_id=K2Y25ECEZ6SEYOV5NPG2EEWFVN5IFFVM54QPK4ZGWZQRNE3I3BCA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.500744707 (87h26m47.371827752s) [noise=true , address=145.79.227.31:4797 , piece_id=AW3Q7GXNWAU5T5Z562KBKXYZPJEL5IRR6XB7EZZPZXUJMBQNOSBA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.501416546 (87h26m47.372499591s) [noise=true , address=145.79.227.16:4909 , piece_id=7TX6RV44XTP4C5DPVEPTSV6ECNGUSXU6G33T6L3AUCFNQLISNHOQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.501422417 (87h26m47.372505462s) [noise=true , address=145.79.227.23:4547 , piece_id=UDL6VDEHGJ6Z4HMRU6HM6XIDUCMSWCEECCUH45FT3FQVLUDOEZGQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.514960194 (87h26m47.386043249s) [noise=true , address=145.79.227.24:4581 , piece_id=WNL2FDFC4N4U5TX522OSVZDEBNO6VVZTL4TOS2C65KHY2CAMHQHA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516209051 (87h26m47.387292086s) [noise=true , address=145.79.227.29:4941 , piece_id=UDD375EXXARYRWOXMWZF25HB7VW4YVWQEFESPW4JNNB4MG7QPPSA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516598713 (87h26m47.387681758s) [noise=true , address=145.79.227.29:4942 , piece_id=SMZGQG7OUYGCHPPJLIS3PH5Y3VU65I46D4NYL2PXCBRSPQRSPZEQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516607911 (87h26m47.387690946s) [seg_pos={0 0}, upload=76331]: Segment reached optimal threshold of 13 pieces. Dec 9 06:55:47.516785880 (87h26m47.387868915s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u): protocol: expected piece hash; context canceled; EOF Dec 9 06:55:47.516824383 (87h26m47.387907448s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Long tail cancellation due to optimal threshold Dec 9 06:55:47.517129214 (87h26m47.388212259s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 06:55:47.517138261 (87h26m47.388221306s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Long tail cancellation due to optimal threshold Dec 9 06:55:47.517375302 (87h26m47.388458337s) [seg_pos={0 0}, upload=76331]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 06:55:47.517466135 (87h26m47.38854917s) [upload=76331]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 06:55:47.517473488 (87h26m47.388556534s) [upload=76331]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:00:17.830318594 (87h31m17.701401639s) [upload=76331]: Done uploading segments... Dec 9 07:00:17.830402994 (87h31m17.70148605s) [upload=76331]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.80663951s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 4.988609912s trace: https://jaeger.dev.storj.io/trace/488f36bcf1592ab1 FAIL (2025-12-09 07:12:51 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.988670578s core uplink eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....X...............X....X.................................] duration: 3.53139567s trace: https://jaeger.dev.storj.io/trace/2c78734d9bae1209 FAIL (2025-12-09 07:12:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.53142183s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [X.........X................X...X...X........................] duration: 5.828647478s trace: https://jaeger.dev.storj.io/trace/5cb4843126cfbf92 FAIL (2025-12-09 07:12:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.828676884s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.......................................X................] duration: 5.665310571s trace: https://jaeger.dev.storj.io/trace/3a1fb019f7c2965d FAIL (2025-12-09 07:12:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 06:55:45.259556946 (87h26m45.130639981s) [upload=76331]: Starting upload Dec 9 06:55:45.259623222 (87h26m45.130706257s) [upload=76331]: Done starting upload Dec 9 06:55:45.259658870 (87h26m45.130741905s) [upload=76331]: Uploading segments... Dec 9 06:55:45.259795500 (87h26m45.130878535s) [upload=76331]: Got next segment. Inline: false Dec 9 06:55:45.259821169 (87h26m45.130904204s) [upload=76331]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 06:55:45.259825137 (87h26m45.130908172s) [upload=76331]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 9 06:55:45.506331314 (87h26m45.377414359s) [seg_pos={0 0}, upload=76331]: Begin upload segment... Dec 9 06:55:45.506397410 (87h26m45.377480445s) [seg_pos={0 0}, upload=76331]: Done begin upload segment. Dec 9 06:55:45.506452925 (87h26m45.37753596s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.506473765 (87h26m45.3775568s) [noise=true , address=77.90.31.15:4698 , piece_id=2Y3GG5Y2QUJ6PFAYODA4V63N4CQXCTLRNH7LISWCXQWOAVCBSULQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507055783 (87h26m45.378138818s) [noise=true , address=145.79.227.41:4086 , piece_id=GXH4GU22OG3BZQSHTK25AHWPFCYKCQEN72NXEPYXO4ZUIW2RSDEA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507493356 (87h26m45.378576391s) [noise=true , address=145.79.227.29:4942 , piece_id=SMZGQG7OUYGCHPPJLIS3PH5Y3VU65I46D4NYL2PXCBRSPQRSPZEQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507543853 (87h26m45.378626898s) [noise=true , address=145.79.227.16:4909 , piece_id=7TX6RV44XTP4C5DPVEPTSV6ECNGUSXU6G33T6L3AUCFNQLISNHOQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507799740 (87h26m45.378882775s) [noise=true , address=145.79.227.29:4941 , piece_id=UDD375EXXARYRWOXMWZF25HB7VW4YVWQEFESPW4JNNB4MG7QPPSA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.507920349 (87h26m45.379003384s) [noise=true , address=145.79.227.31:4797 , piece_id=AW3Q7GXNWAU5T5Z562KBKXYZPJEL5IRR6XB7EZZPZXUJMBQNOSBA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508339767 (87h26m45.379422802s) [noise=true , address=145.79.227.34:4905 , piece_id=RUHC4SR4EO2IF2KEDJNYZOY33EAKF4JR4FTM62NZJCUD2XPO7N6Q, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508566930 (87h26m45.379649965s) [noise=true , address=145.79.227.25:4618 , piece_id=K2Y25ECEZ6SEYOV5NPG2EEWFVN5IFFVM54QPK4ZGWZQRNE3I3BCA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.508739269 (87h26m45.379822314s) [noise=true , address=145.79.227.24:4581 , piece_id=WNL2FDFC4N4U5TX522OSVZDEBNO6VVZTL4TOS2C65KHY2CAMHQHA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509134110 (87h26m45.380217145s) [noise=true , address=145.79.227.24:4582 , piece_id=EAMAU5U3OBF5RA762ZYOARJZ4J2J4MXNEKWKY3JWGT3BX76OB6XQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509258507 (87h26m45.380341543s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509546506 (87h26m45.380629541s) [noise=true , address=145.79.227.21:4473 , piece_id=E4G6ZYPNOHCKQ6FYK4I3Y5V36C7RX5TOKVCQKN2VKHFWV5J62VEA, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.509843851 (87h26m45.380926896s) [noise=true , address=145.79.227.23:4547 , piece_id=UDL6VDEHGJ6Z4HMRU6HM6XIDUCMSWCEECCUH45FT3FQVLUDOEZGQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.510141709 (87h26m45.381224754s) [noise=true , address=145.79.227.38:4977 , piece_id=JZVULFGKR65GE24SFP5U7LHINOZIZ2BXIGQSBTZUSPXEWXEQA32A, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:45.510428084 (87h26m45.381511129s) [noise=true , address=145.79.227.32:4834 , piece_id=N6VJKJMTE5W3HWGCXDGPTLB3BHSFLWBMESU57UOSWPNBM55DZ4MQ, seg_pos={0 0}, upload=76331]: Uploading piece... Dec 9 06:55:46.567032201 (87h26m46.438115236s) [noise=true , address=77.90.31.15:4698 , piece_id=2Y3GG5Y2QUJ6PFAYODA4V63N4CQXCTLRNH7LISWCXQWOAVCBSULQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: storage node overloaded, request limit: 2000 Dec 9 06:55:47.200741008 (87h26m47.071824053s) [upload=76331]: Next returned nil segment Dec 9 06:55:47.200756667 (87h26m47.071839702s) [upload=76331]: Waiting for error group managing segments... Dec 9 06:55:47.488289311 (87h26m47.359372346s) [noise=true , address=145.79.227.41:4086 , piece_id=GXH4GU22OG3BZQSHTK25AHWPFCYKCQEN72NXEPYXO4ZUIW2RSDEA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.488813710 (87h26m47.359896755s) [noise=true , address=145.79.227.21:4473 , piece_id=E4G6ZYPNOHCKQ6FYK4I3Y5V36C7RX5TOKVCQKN2VKHFWV5J62VEA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.488891818 (87h26m47.359974853s) [noise=true , address=145.79.227.38:4977 , piece_id=JZVULFGKR65GE24SFP5U7LHINOZIZ2BXIGQSBTZUSPXEWXEQA32A, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.489842157 (87h26m47.360925202s) [noise=true , address=145.79.227.24:4582 , piece_id=EAMAU5U3OBF5RA762ZYOARJZ4J2J4MXNEKWKY3JWGT3BX76OB6XQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.489996712 (87h26m47.361079747s) [noise=true , address=145.79.227.32:4834 , piece_id=N6VJKJMTE5W3HWGCXDGPTLB3BHSFLWBMESU57UOSWPNBM55DZ4MQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.495771257 (87h26m47.366854302s) [noise=true , address=145.79.227.34:4905 , piece_id=RUHC4SR4EO2IF2KEDJNYZOY33EAKF4JR4FTM62NZJCUD2XPO7N6Q, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.496288763 (87h26m47.367371808s) [noise=true , address=145.79.227.25:4618 , piece_id=K2Y25ECEZ6SEYOV5NPG2EEWFVN5IFFVM54QPK4ZGWZQRNE3I3BCA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.500744707 (87h26m47.371827752s) [noise=true , address=145.79.227.31:4797 , piece_id=AW3Q7GXNWAU5T5Z562KBKXYZPJEL5IRR6XB7EZZPZXUJMBQNOSBA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.501416546 (87h26m47.372499591s) [noise=true , address=145.79.227.16:4909 , piece_id=7TX6RV44XTP4C5DPVEPTSV6ECNGUSXU6G33T6L3AUCFNQLISNHOQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.501422417 (87h26m47.372505462s) [noise=true , address=145.79.227.23:4547 , piece_id=UDL6VDEHGJ6Z4HMRU6HM6XIDUCMSWCEECCUH45FT3FQVLUDOEZGQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.514960194 (87h26m47.386043249s) [noise=true , address=145.79.227.24:4581 , piece_id=WNL2FDFC4N4U5TX522OSVZDEBNO6VVZTL4TOS2C65KHY2CAMHQHA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516209051 (87h26m47.387292086s) [noise=true , address=145.79.227.29:4941 , piece_id=UDD375EXXARYRWOXMWZF25HB7VW4YVWQEFESPW4JNNB4MG7QPPSA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516598713 (87h26m47.387681758s) [noise=true , address=145.79.227.29:4942 , piece_id=SMZGQG7OUYGCHPPJLIS3PH5Y3VU65I46D4NYL2PXCBRSPQRSPZEQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: <nil> Dec 9 06:55:47.516607911 (87h26m47.387690946s) [seg_pos={0 0}, upload=76331]: Segment reached optimal threshold of 13 pieces. Dec 9 06:55:47.516785880 (87h26m47.387868915s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u): protocol: expected piece hash; context canceled; EOF Dec 9 06:55:47.516824383 (87h26m47.387907448s) [noise=true , address=145.79.227.10:4402 , piece_id=DGMGFT3IPDLROHLSCTZGAFFLANSYMO3IQOQLK62AHIAFDRYXBBAA, seg_pos={0 0}, upload=76331]: Long tail cancellation due to optimal threshold Dec 9 06:55:47.517129214 (87h26m47.388212259s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 9 06:55:47.517138261 (87h26m47.388221306s) [noise=true , address=77.90.31.21:4878 , piece_id=LESS3WETF2YGY7LREEGVG6FOAT7ORDRW2ZAJITEHK44ITKCATCBQ, seg_pos={0 0}, upload=76331]: Long tail cancellation due to optimal threshold Dec 9 06:55:47.517375302 (87h26m47.388458337s) [seg_pos={0 0}, upload=76331]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 9 06:55:47.517466135 (87h26m47.38854917s) [upload=76331]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 06:55:47.517473488 (87h26m47.388556534s) [upload=76331]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 07:00:17.830318594 (87h31m17.701401639s) [upload=76331]: Done uploading segments... Dec 9 07:00:17.830402994 (87h31m17.70148605s) [upload=76331]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.665323084s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X...................................X...................] duration: 5.753023204s trace: https://jaeger.dev.storj.io/trace/270563580d82773d FAIL (2025-12-09 07:12:23 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 5.753054642s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......X...............X....X................................] duration: 3.462115437s trace: https://jaeger.dev.storj.io/trace/2c78734d9bae1209 FAIL (2025-12-09 07:12:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.46213834s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X...................................X..................] duration: 13.64042499s trace: https://jaeger.dev.storj.io/trace/270563580d82773d FAIL (2025-12-09 07:12:04 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 13.64045672s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.X.........X................X...X...X.......................] duration: 6.475629884s trace: https://jaeger.dev.storj.io/trace/5cb4843126cfbf92 FAIL (2025-12-09 07:11:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.475671914s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 4.849852579s trace: https://jaeger.dev.storj.io/trace/488f36bcf1592ab1 FAIL (2025-12-09 07:11:36 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.849883538s core uplink eu1

1
...
11630
11631
11632
11633
11634
...
17361