Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 3.051636491s trace: https://jaeger.dev.storj.io/trace/279da5a1e905f33b FAIL (2025-12-08 13:43:22 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 3.051675726s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [.XX.XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXXXXXXX] duration: 5.0013934s trace: https://jaeger.dev.storj.io/trace/3964b83ff46e2e8d FAIL (2025-12-08 13:43:22 UTC) orbiter warning: context deadline exceeded Duration: 5.001412687s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 8.349201276s trace: https://jaeger.dev.storj.io/trace/19a04ed4a9325a12 FAIL (2025-12-08 13:43:08 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 8.349232634s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 4.584373316s trace: https://jaeger.dev.storj.io/trace/279da5a1e905f33b FAIL (2025-12-08 13:43:08 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 4.58440123s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [X.XX.XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXXXXXX] duration: 5.000827106s trace: https://jaeger.dev.storj.io/trace/70c9f40220a27473 FAIL (2025-12-08 13:42:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000853488s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 3.112607189s trace: https://jaeger.dev.storj.io/trace/279da5a1e905f33b FAIL (2025-12-08 13:42:47 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 3.112636866s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 9.064287123s trace: https://jaeger.dev.storj.io/trace/19a04ed4a9325a12 FAIL (2025-12-08 13:42:38 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 9.06431708s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X......X..................................] duration: 10.396352393s trace: https://jaeger.dev.storj.io/trace/26d2fdda31045327 FAIL (2025-12-08 13:42:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 13:15:55.585945137 (69h46m55.457028182s) [upload=61116]: Starting upload Dec 8 13:15:55.586024958 (69h46m55.457108003s) [upload=61116]: Done starting upload Dec 8 13:15:55.586142032 (69h46m55.457225077s) [upload=61116]: Uploading segments... Dec 8 13:15:55.586382891 (69h46m55.457465926s) [upload=61116]: Got next segment. Inline: false Dec 8 13:15:55.586425051 (69h46m55.457508096s) [upload=61116]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 13:15:55.586432224 (69h46m55.457515259s) [upload=61116]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 13:15:55.975069379 (69h46m55.846152414s) [seg_pos={0 0}, upload=61116]: Begin upload segment... Dec 8 13:15:55.975128031 (69h46m55.846211066s) [seg_pos={0 0}, upload=61116]: Done begin upload segment. Dec 8 13:15:55.975176172 (69h46m55.846259217s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975191482 (69h46m55.846274517s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975705481 (69h46m55.846788516s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975788469 (69h46m55.846871504s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976081887 (69h46m55.847164932s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976365817 (69h46m55.847448862s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976605644 (69h46m55.847688679s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977022939 (69h46m55.848105974s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977329733 (69h46m55.848412778s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977433501 (69h46m55.848516536s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977759893 (69h46m55.848842928s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978113566 (69h46m55.849196601s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978477388 (69h46m55.849560424s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978716584 (69h46m55.849799629s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.979021695 (69h46m55.85010473s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.979276540 (69h46m55.850359575s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:57.135990914 (69h46m57.007073969s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 13:15:57.854764714 (69h46m57.725847759s) [upload=61116]: Next returned nil segment Dec 8 13:15:57.854780914 (69h46m57.725863949s) [upload=61116]: Waiting for error group managing segments... Dec 8 13:15:58.152891657 (69h46m58.023974702s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.344029421 (69h46m58.215112456s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.878402214 (69h46m58.749485249s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.882644703 (69h46m58.753727738s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.901677445 (69h46m58.77276048s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.913588956 (69h46m58.784671991s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.053655972 (69h46m58.924739007s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.104497705 (69h46m58.97558074s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.108923833 (69h46m58.980006868s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.110716147 (69h46m58.981799182s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.113697604 (69h46m58.984780639s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.116192063 (69h46m58.987275098s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.127043066 (69h46m58.998126111s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.127068845 (69h46m58.99815192s) [seg_pos={0 0}, upload=61116]: Segment reached optimal threshold of 13 pieces. Dec 8 13:15:59.127555342 (69h46m58.998638387s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: expected piece hash; context canceled; EOF Dec 8 13:15:59.127563667 (69h46m58.998646712s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold Dec 8 13:15:59.127970713 (69h46m58.999053758s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Dec 8 13:15:59.127989598 (69h46m58.999072633s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold Dec 8 13:15:59.128068279 (69h46m58.999151314s) [seg_pos={0 0}, upload=61116]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 13:15:59.128237932 (69h46m58.999320967s) [upload=61116]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 13:15:59.128244734 (69h46m58.99932777s) [upload=61116]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 13:17:38.673966206 (69h48m38.545049271s) [upload=61116]: Done uploading segments... Dec 8 13:17:38.674070214 (69h48m38.545153249s) [upload=61116]: Upload finished. err: metaclient: context deadline exceeded Duration: 10.396382281s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [........X.....X.....XX.XX.X............X.......XXX..XX....X.] duration: 16.366880967s trace: https://jaeger.dev.storj.io/trace/3e69d589b2720cd4 FAIL (2025-12-08 13:42:25 UTC) orbiter warning: context deadline exceeded Duration: 16.366927936s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [........X...X.X.....X........X......X....X...............X..] duration: 10.941345759s trace: https://jaeger.dev.storj.io/trace/77e4a1f8db8dca4b FAIL (2025-12-08 13:42:21 UTC) orbiter warning: context deadline exceeded Duration: 10.941379653s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.........X...X.X.....X........X......X....X...............X.] duration: 26.025274327s trace: https://jaeger.dev.storj.io/trace/77e4a1f8db8dca4b FAIL (2025-12-08 13:42:09 UTC) orbiter warning: context deadline exceeded Duration: 26.025288835s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [.........X.....X.....XX.XX.X............X.......XXX..XX....X] duration: 1m0.000794989s trace: https://jaeger.dev.storj.io/trace/3e69d589b2720cd4 FAIL (2025-12-08 13:42:08 UTC) orbiter warning: context deadline exceeded Duration: 1m0.000929435s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 56 out of 60 is failed (threshold=6) [XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXXXXX] duration: 5.001085761s trace: https://jaeger.dev.storj.io/trace/42210bd2d627558a FAIL (2025-12-08 13:41:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001109668s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 7.881201568s trace: https://jaeger.dev.storj.io/trace/19a04ed4a9325a12 FAIL (2025-12-08 13:41:44 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.881221925s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..........X...X.X.....X........X......X....X...............X] duration: 30.00099742s trace: https://jaeger.dev.storj.io/trace/77e4a1f8db8dca4b FAIL (2025-12-08 13:41:43 UTC) orbiter warning: context deadline exceeded Duration: 30.001025423s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 3.054613517s trace: https://jaeger.dev.storj.io/trace/279da5a1e905f33b FAIL (2025-12-08 13:41:39 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 3.054644075s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X......X.................................] duration: 11.086780399s trace: https://jaeger.dev.storj.io/trace/26d2fdda31045327 FAIL (2025-12-08 13:41:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 13:15:55.585945137 (69h46m55.457028182s) [upload=61116]: Starting upload Dec 8 13:15:55.586024958 (69h46m55.457108003s) [upload=61116]: Done starting upload Dec 8 13:15:55.586142032 (69h46m55.457225077s) [upload=61116]: Uploading segments... Dec 8 13:15:55.586382891 (69h46m55.457465926s) [upload=61116]: Got next segment. Inline: false Dec 8 13:15:55.586425051 (69h46m55.457508096s) [upload=61116]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 13:15:55.586432224 (69h46m55.457515259s) [upload=61116]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 13:15:55.975069379 (69h46m55.846152414s) [seg_pos={0 0}, upload=61116]: Begin upload segment... Dec 8 13:15:55.975128031 (69h46m55.846211066s) [seg_pos={0 0}, upload=61116]: Done begin upload segment. Dec 8 13:15:55.975176172 (69h46m55.846259217s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975191482 (69h46m55.846274517s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975705481 (69h46m55.846788516s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975788469 (69h46m55.846871504s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976081887 (69h46m55.847164932s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976365817 (69h46m55.847448862s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976605644 (69h46m55.847688679s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977022939 (69h46m55.848105974s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977329733 (69h46m55.848412778s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977433501 (69h46m55.848516536s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977759893 (69h46m55.848842928s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978113566 (69h46m55.849196601s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978477388 (69h46m55.849560424s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978716584 (69h46m55.849799629s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.979021695 (69h46m55.85010473s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.979276540 (69h46m55.850359575s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:57.135990914 (69h46m57.007073969s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 13:15:57.854764714 (69h46m57.725847759s) [upload=61116]: Next returned nil segment Dec 8 13:15:57.854780914 (69h46m57.725863949s) [upload=61116]: Waiting for error group managing segments... Dec 8 13:15:58.152891657 (69h46m58.023974702s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.344029421 (69h46m58.215112456s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.878402214 (69h46m58.749485249s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.882644703 (69h46m58.753727738s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.901677445 (69h46m58.77276048s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.913588956 (69h46m58.784671991s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.053655972 (69h46m58.924739007s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.104497705 (69h46m58.97558074s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.108923833 (69h46m58.980006868s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.110716147 (69h46m58.981799182s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.113697604 (69h46m58.984780639s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.116192063 (69h46m58.987275098s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.127043066 (69h46m58.998126111s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.127068845 (69h46m58.99815192s) [seg_pos={0 0}, upload=61116]: Segment reached optimal threshold of 13 pieces. Dec 8 13:15:59.127555342 (69h46m58.998638387s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: expected piece hash; context canceled; EOF Dec 8 13:15:59.127563667 (69h46m58.998646712s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold Dec 8 13:15:59.127970713 (69h46m58.999053758s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Dec 8 13:15:59.127989598 (69h46m58.999072633s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold Dec 8 13:15:59.128068279 (69h46m58.999151314s) [seg_pos={0 0}, upload=61116]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 13:15:59.128237932 (69h46m58.999320967s) [upload=61116]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 13:15:59.128244734 (69h46m58.99932777s) [upload=61116]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 13:17:38.673966206 (69h48m38.545049271s) [upload=61116]: Done uploading segments... Dec 8 13:17:38.674070214 (69h48m38.545153249s) [upload=61116]: Upload finished. err: metaclient: context deadline exceeded Duration: 11.086810756s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 8.06277239s trace: https://jaeger.dev.storj.io/trace/19a04ed4a9325a12 FAIL (2025-12-08 13:41:25 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 8.062801043s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 55 out of 60 is failed (threshold=6) [.XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXXXX] duration: 5.000913034s trace: https://jaeger.dev.storj.io/trace/984ebc0b7a0f0a2 FAIL (2025-12-08 13:41:19 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000941248s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X......X................................] duration: 20.37515338s trace: https://jaeger.dev.storj.io/trace/26d2fdda31045327 FAIL (2025-12-08 13:41:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 13:15:55.585945137 (69h46m55.457028182s) [upload=61116]: Starting upload Dec 8 13:15:55.586024958 (69h46m55.457108003s) [upload=61116]: Done starting upload Dec 8 13:15:55.586142032 (69h46m55.457225077s) [upload=61116]: Uploading segments... Dec 8 13:15:55.586382891 (69h46m55.457465926s) [upload=61116]: Got next segment. Inline: false Dec 8 13:15:55.586425051 (69h46m55.457508096s) [upload=61116]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 13:15:55.586432224 (69h46m55.457515259s) [upload=61116]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 13:15:55.975069379 (69h46m55.846152414s) [seg_pos={0 0}, upload=61116]: Begin upload segment... Dec 8 13:15:55.975128031 (69h46m55.846211066s) [seg_pos={0 0}, upload=61116]: Done begin upload segment. Dec 8 13:15:55.975176172 (69h46m55.846259217s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975191482 (69h46m55.846274517s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975705481 (69h46m55.846788516s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975788469 (69h46m55.846871504s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976081887 (69h46m55.847164932s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976365817 (69h46m55.847448862s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976605644 (69h46m55.847688679s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977022939 (69h46m55.848105974s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977329733 (69h46m55.848412778s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977433501 (69h46m55.848516536s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977759893 (69h46m55.848842928s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978113566 (69h46m55.849196601s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978477388 (69h46m55.849560424s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978716584 (69h46m55.849799629s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.979021695 (69h46m55.85010473s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.979276540 (69h46m55.850359575s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:57.135990914 (69h46m57.007073969s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 13:15:57.854764714 (69h46m57.725847759s) [upload=61116]: Next returned nil segment Dec 8 13:15:57.854780914 (69h46m57.725863949s) [upload=61116]: Waiting for error group managing segments... Dec 8 13:15:58.152891657 (69h46m58.023974702s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.344029421 (69h46m58.215112456s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.878402214 (69h46m58.749485249s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.882644703 (69h46m58.753727738s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.901677445 (69h46m58.77276048s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.913588956 (69h46m58.784671991s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.053655972 (69h46m58.924739007s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.104497705 (69h46m58.97558074s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.108923833 (69h46m58.980006868s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.110716147 (69h46m58.981799182s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.113697604 (69h46m58.984780639s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.116192063 (69h46m58.987275098s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.127043066 (69h46m58.998126111s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.127068845 (69h46m58.99815192s) [seg_pos={0 0}, upload=61116]: Segment reached optimal threshold of 13 pieces. Dec 8 13:15:59.127555342 (69h46m58.998638387s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: expected piece hash; context canceled; EOF Dec 8 13:15:59.127563667 (69h46m58.998646712s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold Dec 8 13:15:59.127970713 (69h46m58.999053758s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Dec 8 13:15:59.127989598 (69h46m58.999072633s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold Dec 8 13:15:59.128068279 (69h46m58.999151314s) [seg_pos={0 0}, upload=61116]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 13:15:59.128237932 (69h46m58.999320967s) [upload=61116]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 13:15:59.128244734 (69h46m58.99932777s) [upload=61116]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 13:17:38.673966206 (69h48m38.545049271s) [upload=61116]: Done uploading segments... Dec 8 13:17:38.674070214 (69h48m38.545153249s) [upload=61116]: Upload finished. err: metaclient: context deadline exceeded Duration: 20.375172507s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 3.016329358s trace: https://jaeger.dev.storj.io/trace/279da5a1e905f33b FAIL (2025-12-08 13:41:09 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 3.016358944s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 10.926036284s trace: https://jaeger.dev.storj.io/trace/19a04ed4a9325a12 FAIL (2025-12-08 13:41:00 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 10.926065168s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 54 out of 60 is failed (threshold=6) [..XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXXX] duration: 5.00123951s trace: https://jaeger.dev.storj.io/trace/4fd0603c1dcb50cd FAIL (2025-12-08 13:40:42 UTC) orbiter warning: context deadline exceeded Duration: 5.001253356s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 9.340982664s trace: https://jaeger.dev.storj.io/trace/19a04ed4a9325a12 FAIL (2025-12-08 13:40:39 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 9.341015437s core uplink us1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 466.366475ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6e6474fd8c311e42 FAIL (2025-12-08 13:40:36 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 466.391452ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [X.........X.....X.....XX.XX.X............X.......XXX..XX....] duration: 32.465069133s trace: https://jaeger.dev.storj.io/trace/7b4b153604ecdeab FAIL (2025-12-08 13:40:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 32.465101765s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 3.988041299s trace: https://jaeger.dev.storj.io/trace/279da5a1e905f33b FAIL (2025-12-08 13:40:28 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 3.988072647s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X......X...............................] duration: 23.778254309s trace: https://jaeger.dev.storj.io/trace/26d2fdda31045327 FAIL (2025-12-08 13:40:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 13:15:55.585945137 (69h46m55.457028182s) [upload=61116]: Starting upload Dec 8 13:15:55.586024958 (69h46m55.457108003s) [upload=61116]: Done starting upload Dec 8 13:15:55.586142032 (69h46m55.457225077s) [upload=61116]: Uploading segments... Dec 8 13:15:55.586382891 (69h46m55.457465926s) [upload=61116]: Got next segment. Inline: false Dec 8 13:15:55.586425051 (69h46m55.457508096s) [upload=61116]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 13:15:55.586432224 (69h46m55.457515259s) [upload=61116]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 13:15:55.975069379 (69h46m55.846152414s) [seg_pos={0 0}, upload=61116]: Begin upload segment... Dec 8 13:15:55.975128031 (69h46m55.846211066s) [seg_pos={0 0}, upload=61116]: Done begin upload segment. Dec 8 13:15:55.975176172 (69h46m55.846259217s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975191482 (69h46m55.846274517s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975705481 (69h46m55.846788516s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975788469 (69h46m55.846871504s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976081887 (69h46m55.847164932s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976365817 (69h46m55.847448862s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976605644 (69h46m55.847688679s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977022939 (69h46m55.848105974s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977329733 (69h46m55.848412778s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977433501 (69h46m55.848516536s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977759893 (69h46m55.848842928s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978113566 (69h46m55.849196601s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978477388 (69h46m55.849560424s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978716584 (69h46m55.849799629s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.979021695 (69h46m55.85010473s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.979276540 (69h46m55.850359575s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:57.135990914 (69h46m57.007073969s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 13:15:57.854764714 (69h46m57.725847759s) [upload=61116]: Next returned nil segment Dec 8 13:15:57.854780914 (69h46m57.725863949s) [upload=61116]: Waiting for error group managing segments... Dec 8 13:15:58.152891657 (69h46m58.023974702s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.344029421 (69h46m58.215112456s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.878402214 (69h46m58.749485249s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.882644703 (69h46m58.753727738s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.901677445 (69h46m58.77276048s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.913588956 (69h46m58.784671991s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.053655972 (69h46m58.924739007s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.104497705 (69h46m58.97558074s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.108923833 (69h46m58.980006868s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.110716147 (69h46m58.981799182s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.113697604 (69h46m58.984780639s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.116192063 (69h46m58.987275098s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.127043066 (69h46m58.998126111s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.127068845 (69h46m58.99815192s) [seg_pos={0 0}, upload=61116]: Segment reached optimal threshold of 13 pieces. Dec 8 13:15:59.127555342 (69h46m58.998638387s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: expected piece hash; context canceled; EOF Dec 8 13:15:59.127563667 (69h46m58.998646712s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold Dec 8 13:15:59.127970713 (69h46m58.999053758s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Dec 8 13:15:59.127989598 (69h46m58.999072633s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold Dec 8 13:15:59.128068279 (69h46m58.999151314s) [seg_pos={0 0}, upload=61116]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 13:15:59.128237932 (69h46m58.999320967s) [upload=61116]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 13:15:59.128244734 (69h46m58.99932777s) [upload=61116]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 13:17:38.673966206 (69h48m38.545049271s) [upload=61116]: Done uploading segments... Dec 8 13:17:38.674070214 (69h48m38.545153249s) [upload=61116]: Upload finished. err: metaclient: context deadline exceeded Duration: 23.778311566s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........X...X.X.....X........X......X....X...............] duration: 18.785707121s trace: https://jaeger.dev.storj.io/trace/341ce6a64bf5a159 FAIL (2025-12-08 13:40:23 UTC) orbiter warning: context deadline exceeded Duration: 18.785763417s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 53 out of 60 is failed (threshold=6) [...XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXXX] duration: 5.0013255s trace: https://jaeger.dev.storj.io/trace/774e54cea65a5dd FAIL (2025-12-08 13:40:15 UTC) orbiter warning: context deadline exceeded Duration: 5.001349335s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [............X...X.X.....X........X......X....X..............] duration: 11.953726695s trace: https://jaeger.dev.storj.io/trace/341ce6a64bf5a159 FAIL (2025-12-08 13:40:00 UTC) orbiter warning: context deadline exceeded Duration: 11.953753566s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [.X.........X.....X.....XX.XX.X............X.......XXX..XX...] duration: 16.962037759s trace: https://jaeger.dev.storj.io/trace/7b4b153604ecdeab FAIL (2025-12-08 13:39:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.962066633s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 8.246704502s trace: https://jaeger.dev.storj.io/trace/19a04ed4a9325a12 FAIL (2025-12-08 13:39:53 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 8.246768803s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X......X..............................] duration: 24.339358373s trace: https://jaeger.dev.storj.io/trace/26d2fdda31045327 FAIL (2025-12-08 13:39:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 13:15:55.585945137 (69h46m55.457028182s) [upload=61116]: Starting upload Dec 8 13:15:55.586024958 (69h46m55.457108003s) [upload=61116]: Done starting upload Dec 8 13:15:55.586142032 (69h46m55.457225077s) [upload=61116]: Uploading segments... Dec 8 13:15:55.586382891 (69h46m55.457465926s) [upload=61116]: Got next segment. Inline: false Dec 8 13:15:55.586425051 (69h46m55.457508096s) [upload=61116]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 13:15:55.586432224 (69h46m55.457515259s) [upload=61116]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 13:15:55.975069379 (69h46m55.846152414s) [seg_pos={0 0}, upload=61116]: Begin upload segment... Dec 8 13:15:55.975128031 (69h46m55.846211066s) [seg_pos={0 0}, upload=61116]: Done begin upload segment. Dec 8 13:15:55.975176172 (69h46m55.846259217s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975191482 (69h46m55.846274517s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975705481 (69h46m55.846788516s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975788469 (69h46m55.846871504s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976081887 (69h46m55.847164932s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976365817 (69h46m55.847448862s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976605644 (69h46m55.847688679s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977022939 (69h46m55.848105974s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977329733 (69h46m55.848412778s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977433501 (69h46m55.848516536s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977759893 (69h46m55.848842928s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978113566 (69h46m55.849196601s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978477388 (69h46m55.849560424s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978716584 (69h46m55.849799629s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.979021695 (69h46m55.85010473s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.979276540 (69h46m55.850359575s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:57.135990914 (69h46m57.007073969s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 13:15:57.854764714 (69h46m57.725847759s) [upload=61116]: Next returned nil segment Dec 8 13:15:57.854780914 (69h46m57.725863949s) [upload=61116]: Waiting for error group managing segments... Dec 8 13:15:58.152891657 (69h46m58.023974702s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.344029421 (69h46m58.215112456s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.878402214 (69h46m58.749485249s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.882644703 (69h46m58.753727738s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.901677445 (69h46m58.77276048s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.913588956 (69h46m58.784671991s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.053655972 (69h46m58.924739007s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.104497705 (69h46m58.97558074s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.108923833 (69h46m58.980006868s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.110716147 (69h46m58.981799182s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.113697604 (69h46m58.984780639s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.116192063 (69h46m58.987275098s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.127043066 (69h46m58.998126111s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.127068845 (69h46m58.99815192s) [seg_pos={0 0}, upload=61116]: Segment reached optimal threshold of 13 pieces. Dec 8 13:15:59.127555342 (69h46m58.998638387s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: expected piece hash; context canceled; EOF Dec 8 13:15:59.127563667 (69h46m58.998646712s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold Dec 8 13:15:59.127970713 (69h46m58.999053758s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Dec 8 13:15:59.127989598 (69h46m58.999072633s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold Dec 8 13:15:59.128068279 (69h46m58.999151314s) [seg_pos={0 0}, upload=61116]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 13:15:59.128237932 (69h46m58.999320967s) [upload=61116]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 13:15:59.128244734 (69h46m58.99932777s) [upload=61116]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 13:17:38.673966206 (69h48m38.545049271s) [upload=61116]: Done uploading segments... Dec 8 13:17:38.674070214 (69h48m38.545153249s) [upload=61116]: Upload finished. err: metaclient: context deadline exceeded Duration: 24.339389142s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.888648977s trace: https://jaeger.dev.storj.io/trace/279da5a1e905f33b FAIL (2025-12-08 13:39:41 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.888678975s core uplink us1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 468.124103ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6e6474fd8c311e42 FAIL (2025-12-08 13:39:40 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 468.144121ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 52 out of 60 is failed (threshold=6) [....XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XXXX] duration: 5.001597003s trace: https://jaeger.dev.storj.io/trace/820fc8519e2b84 FAIL (2025-12-08 13:39:38 UTC) orbiter warning: context deadline exceeded Duration: 5.001610359s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 472.54438ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6e6474fd8c311e42 FAIL (2025-12-08 13:39:24 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 472.598753ms core satellite eu1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.............X...X.X.....X........X......X....X.............] duration: 24.208793486s trace: https://jaeger.dev.storj.io/trace/341ce6a64bf5a159 FAIL (2025-12-08 13:39:16 UTC) orbiter warning: context deadline exceeded Duration: 24.208823034s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [..X.........X.....X.....XX.XX.X............X.......XXX..XX..] duration: 19.013279597s trace: https://jaeger.dev.storj.io/trace/7b4b153604ecdeab FAIL (2025-12-08 13:38:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.013309124s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 7.693718413s trace: https://jaeger.dev.storj.io/trace/19a04ed4a9325a12 FAIL (2025-12-08 13:38:43 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 7.693744142s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.905835576s trace: https://jaeger.dev.storj.io/trace/279da5a1e905f33b FAIL (2025-12-08 13:38:39 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.905864781s core uplink us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..............X...X.X.....X........X......X....X............] duration: 25.471746443s trace: https://jaeger.dev.storj.io/trace/341ce6a64bf5a159 FAIL (2025-12-08 13:38:37 UTC) orbiter warning: context deadline exceeded Duration: 25.471759548s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 52 out of 60 is failed (threshold=6) [X....XX.XX.XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX.XXX] duration: 5.00101961s trace: https://jaeger.dev.storj.io/trace/47a935113034dd00 FAIL (2025-12-08 13:38:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001043455s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X......X.............................] duration: 8.877395504s trace: https://jaeger.dev.storj.io/trace/26d2fdda31045327 FAIL (2025-12-08 13:38:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 8 13:15:55.585945137 (69h46m55.457028182s) [upload=61116]: Starting upload Dec 8 13:15:55.586024958 (69h46m55.457108003s) [upload=61116]: Done starting upload Dec 8 13:15:55.586142032 (69h46m55.457225077s) [upload=61116]: Uploading segments... Dec 8 13:15:55.586382891 (69h46m55.457465926s) [upload=61116]: Got next segment. Inline: false Dec 8 13:15:55.586425051 (69h46m55.457508096s) [upload=61116]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 8 13:15:55.586432224 (69h46m55.457515259s) [upload=61116]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 8 13:15:55.975069379 (69h46m55.846152414s) [seg_pos={0 0}, upload=61116]: Begin upload segment... Dec 8 13:15:55.975128031 (69h46m55.846211066s) [seg_pos={0 0}, upload=61116]: Done begin upload segment. Dec 8 13:15:55.975176172 (69h46m55.846259217s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975191482 (69h46m55.846274517s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975705481 (69h46m55.846788516s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.975788469 (69h46m55.846871504s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976081887 (69h46m55.847164932s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976365817 (69h46m55.847448862s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.976605644 (69h46m55.847688679s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977022939 (69h46m55.848105974s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977329733 (69h46m55.848412778s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977433501 (69h46m55.848516536s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.977759893 (69h46m55.848842928s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978113566 (69h46m55.849196601s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978477388 (69h46m55.849560424s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.978716584 (69h46m55.849799629s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.979021695 (69h46m55.85010473s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:55.979276540 (69h46m55.850359575s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Uploading piece... Dec 8 13:15:57.135990914 (69h46m57.007073969s) [noise=true , address=77.90.31.15:4699 , piece_id=OFHPJKA7EIH2XFPQXEURZKGKRIZZ6ASAFGZM2CSXKFUIOIOKYUXQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: storage node overloaded, request limit: 2000 Dec 8 13:15:57.854764714 (69h46m57.725847759s) [upload=61116]: Next returned nil segment Dec 8 13:15:57.854780914 (69h46m57.725863949s) [upload=61116]: Waiting for error group managing segments... Dec 8 13:15:58.152891657 (69h46m58.023974702s) [noise=true , address=77.90.31.10:4520 , piece_id=B7BUGGRYR7OHNABG2NZKJRKIQ6OQAMTO3V35YAPJ4Y5KUIVZ4IPA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.344029421 (69h46m58.215112456s) [noise=true , address=77.90.31.11:4554 , piece_id=M4DSLQOFVNGPDCC7HTDZNBZXXFTMOFSBX2TO2JWCI67GPKX2CX3A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.878402214 (69h46m58.749485249s) [noise=true , address=145.79.227.28:4727 , piece_id=TZ3WVWHG2O7TOVUOK5CSQR3K5OO46APC3HE4UAY2YSLDVX6YPRZA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.882644703 (69h46m58.753727738s) [noise=true , address=145.79.227.41:4085 , piece_id=5ZCDZYZCTTQBULKPPJKZY3JH7KX5DHXWICJRTSSZZI3WPOBQWVQQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.901677445 (69h46m58.77276048s) [noise=true , address=145.79.227.22:4510 , piece_id=U5IDN3J7L4F3K7G6MCTZCPP37INHYV23KSB5H77RGMV5Q4YOIFAQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:58.913588956 (69h46m58.784671991s) [noise=true , address=145.79.227.34:4906 , piece_id=PJXFHSDWDCF4A5LJUKBENGBDIJUEUBSNXXYWXWKOU4YJQDR54NJQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.053655972 (69h46m58.924739007s) [noise=true , address=145.79.227.27:4689 , piece_id=TEYMEW3JF3DAMTFJUC4XTKZDAN5C5CVJBE553UNPBNCTKBKTUBEQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.104497705 (69h46m58.97558074s) [noise=true , address=145.79.227.38:4978 , piece_id=5S4MMFGEJKTJKDNFZSBP6VVQONSZTECUW7URTDBF4VK4LHQ3D65A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.108923833 (69h46m58.980006868s) [noise=true , address=145.79.227.9:4365 , piece_id=HX5ZPWPFGOR6GB4JUKVBODTDRG3DC4VV52BSIM7ATT7TLM5HWDJA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.110716147 (69h46m58.981799182s) [noise=true , address=145.79.227.43:4158 , piece_id=TQPZAKMXN6MQZ37FGLLCSCOH25UCBUEGMVNXXFKOTTMTHCVKPFHA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.113697604 (69h46m58.984780639s) [noise=true , address=145.79.227.43:4157 , piece_id=5PDQS2PMZ2KLSPLUI5C3SYNZH2RYCZBPBCHRIQYAE7HF6IXMH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.116192063 (69h46m58.987275098s) [noise=true , address=145.79.227.20:4439 , piece_id=HTNEPA6SV4IE6FV5X5OW3TSP3VGOUHAP5UJRU7PDYDOBXA37DY7A, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.127043066 (69h46m58.998126111s) [noise=true , address=145.79.227.39:4014 , piece_id=LN55VHVLXYHL4AKJO6CSM3QBB2A652TDXFTXJBRNHTESAVM4QALQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: <nil> Dec 8 13:15:59.127068845 (69h46m58.99815192s) [seg_pos={0 0}, upload=61116]: Segment reached optimal threshold of 13 pieces. Dec 8 13:15:59.127555342 (69h46m58.998638387s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW): protocol: expected piece hash; context canceled; EOF Dec 8 13:15:59.127563667 (69h46m58.998646712s) [noise=true , address=145.79.227.23:4547 , piece_id=6BPYIMCXQU6LRP67RTJ4CSZ75JOX6GCBBE5UOU3IUY5W3MPNH6SA, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold Dec 8 13:15:59.127970713 (69h46m58.999053758s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Dec 8 13:15:59.127989598 (69h46m58.999072633s) [noise=true , address=145.79.227.25:4617 , piece_id=UOFGEGW6BL7K7UNTTCSXPAF3WYGOQ2JNMK62ONQDUFBB22NFR2NQ, seg_pos={0 0}, upload=61116]: Long tail cancellation due to optimal threshold Dec 8 13:15:59.128068279 (69h46m58.999151314s) [seg_pos={0 0}, upload=61116]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 8 13:15:59.128237932 (69h46m58.999320967s) [upload=61116]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 8 13:15:59.128244734 (69h46m58.99932777s) [upload=61116]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 8 13:17:38.673966206 (69h48m38.545049271s) [upload=61116]: Done uploading segments... Dec 8 13:17:38.674070214 (69h48m38.545153249s) [upload=61116]: Upload finished. err: metaclient: context deadline exceeded Duration: 8.877421703s core uplink us-select-1
checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 469.212976ms remote: "tcp:35.207.121.91:443" target: https://eu1.storj.io/login trace: https://jaeger.dev.storj.io/trace/6e6474fd8c311e42 FAIL (2025-12-08 13:38:17 UTC) orbiter warning: Get "https://eu1.storj.io/login": EOF Duration: 469.243084ms core satellite eu1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [...X.........X.....X.....XX.XX.X............X.......XXX..XX.] duration: 25.679492191s trace: https://jaeger.dev.storj.io/trace/7b4b153604ecdeab FAIL (2025-12-08 13:38:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 25.67951158s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.869425751s trace: https://jaeger.dev.storj.io/trace/279da5a1e905f33b FAIL (2025-12-08 13:38:00 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.869453893s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.000717544s trace: https://jaeger.dev.storj.io/trace/279da5a1e905f33b FAIL (2025-12-08 13:37:55 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000739646s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 9.430260266s trace: https://jaeger.dev.storj.io/trace/19a04ed4a9325a12 FAIL (2025-12-08 13:37:52 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 9.430289714s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://eu1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.eu1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)