Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [.........X............X.X.XX.XX.XX..X.....XX...X.X.X.X....XX] duration:
30.00086349s trace:
https://jaeger.dev.storj.io/trace/34dcbf1241a033f7
FAIL (2025-12-08 15:44:53 UTC)
orbiter warning: context deadline exceeded
Duration: 30.000888989s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
6.261048917s trace:
https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182
FAIL (2025-12-08 15:44:49 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 6.261078033s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [............................X.........X...X..............X..] duration:
7.250892905s trace:
https://jaeger.dev.storj.io/trace/3711a2689ed652fd
FAIL (2025-12-08 15:44:30 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: VKXVHWX7LFKBH4E32F6CMAHMQX2P5AREIEIWHTXK7U22TEWDMRFQ) context deadline exceeded; piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: AUU7NTJQWR52XFWHVONDFHZHKXPOUS2W4UL4OF4XGTWV4ABETSHQ) context deadline exceeded; piecestore: (Node ID: 12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, Piece ID: 2AI2DCS6TNPJGURTBMNDKT5N3GISWDVOC5VTLIP6MVUAEWP2V3ZQ) context deadline exceeded
======LOGS======
Dec 8 15:43:33.737263649 (72h14m33.608346694s) [upload=63248]: Starting upload
Dec 8 15:43:33.737410899 (72h14m33.608493944s) [upload=63248]: Done starting upload
Dec 8 15:43:33.737492094 (72h14m33.608575149s) [upload=63248]: Uploading segments...
Dec 8 15:43:33.739621138 (72h14m33.610704183s) [upload=63248]: Got next segment. Inline: false
Dec 8 15:43:33.739675270 (72h14m33.610758305s) [upload=63248]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:43:33.739681903 (72h14m33.610764938s) [upload=63248]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:43:33.988217234 (72h14m33.859300269s) [seg_pos={0 0}, upload=63248]: Begin upload segment...
Dec 8 15:43:33.988366548 (72h14m33.859449583s) [seg_pos={0 0}, upload=63248]: Done begin upload segment.
Dec 8 15:43:33.988395153 (72h14m33.859478198s) [noise=true , address=77.90.31.19:4842 , piece_id=D6RHPHJCJTV7I6UF5HQSPFFNVVHJDYXOIY3C747PLFFACW32INWA, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.988434698 (72h14m33.859517763s) [noise=true , address=145.79.227.27:4689 , piece_id=LAMO54F6RV2YAJQNHCIOJNSAQGCXK23WHVJJ6VC2REYJ7LUW5R3Q, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.989099694 (72h14m33.860182729s) [noise=true , address=145.79.227.29:4943 , piece_id=TJFW4JKEHHCDTMPVS46BI4OYF35BVHE3VIFM6LD2CJKRFCZXL5FA, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.989662838 (72h14m33.860745873s) [noise=true , address=145.79.227.22:4510 , piece_id=BVTVWVC7VRYL7TREJ63BRWY6LAHU2N5AHYJR3JRHQPPEEITTMFVQ, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.990143793 (72h14m33.861226828s) [noise=true , address=145.79.227.43:4157 , piece_id=3P7NOOCSNAUBKH4SVL2SUSQOADSA3IKRHTPRL4P7VGH62DL23LFQ, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.990589972 (72h14m33.861673007s) [noise=true , address=77.90.31.10:4519 , piece_id=PGLC7MAH7IJ7GP6DTN65WUESDI3Y2JNDRVFR3QJJBNZV2H2SFJSA, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.991038847 (72h14m33.862121882s) [noise=true , address=145.79.227.39:4015 , piece_id=AUU7NTJQWR52XFWHVONDFHZHKXPOUS2W4UL4OF4XGTWV4ABETSHQ, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.991412218 (72h14m33.862495263s) [noise=true , address=145.79.227.38:4978 , piece_id=2AI2DCS6TNPJGURTBMNDKT5N3GISWDVOC5VTLIP6MVUAEWP2V3ZQ, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.991728711 (72h14m33.862811746s) [noise=true , address=145.79.227.38:4979 , piece_id=4A4TZ2YT5KBQVYX7XSKIUMFJBLR3R7NJOZCVXEVVIK67VC744VXA, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.992474100 (72h14m33.863557145s) [noise=true , address=145.79.227.21:4475 , piece_id=PLY2YAALNNQKXXF5MZUUKDUV4HP7XUDRD3QXJTTKQI6FCGLHEA7A, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.992900823 (72h14m33.863983858s) [noise=true , address=145.79.227.28:4727 , piece_id=BSP57B4M3L7UIEDLVM2ASNQRK5BYLGTMAJ633QN4WFNS3OCNWDGQ, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.993301476 (72h14m33.864384521s) [noise=true , address=145.79.227.41:4085 , piece_id=ODJABK4R6SHNXNNQ47WI7W4YB4NR5A6QDTUE4WXVUFBVE7X3AJCQ, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.993926856 (72h14m33.865009922s) [noise=true , address=145.79.227.25:4619 , piece_id=CJJ7CONWO42FFSV7QMB4JSATW7BUDIS6KJXYX2SIBQMWE3UTYA4A, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.994260803 (72h14m33.865343838s) [noise=true , address=77.90.31.14:4661 , piece_id=BON7N46ZON2BYBD4UHNHPAECUF2OL7KV6B7Z2SZWVS6OGTEAK3RA, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.994756687 (72h14m33.865839732s) [noise=true , address=145.79.227.31:4799 , piece_id=VKXVHWX7LFKBH4E32F6CMAHMQX2P5AREIEIWHTXK7U22TEWDMRFQ, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.995123304 (72h14m33.866206349s) [noise=true , address=145.79.227.26:4653 , piece_id=LMBLPSOTMKPFS4KGAJOSRR2EM2MVKJIXSRE5MIB4SF552S2O3QYQ, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:35.816987512 (72h14m35.688070547s) [upload=63248]: Next returned nil segment
Dec 8 15:43:35.817000286 (72h14m35.688083331s) [upload=63248]: Waiting for error group managing segments...
Dec 8 15:43:36.138037836 (72h14m36.009120871s) [noise=true , address=77.90.31.10:4519 , piece_id=PGLC7MAH7IJ7GP6DTN65WUESDI3Y2JNDRVFR3QJJBNZV2H2SFJSA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:36.161877783 (72h14m36.032960818s) [noise=true , address=145.79.227.38:4979 , piece_id=4A4TZ2YT5KBQVYX7XSKIUMFJBLR3R7NJOZCVXEVVIK67VC744VXA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:36.164129480 (72h14m36.035212525s) [noise=true , address=145.79.227.31:4799 , piece_id=VKXVHWX7LFKBH4E32F6CMAHMQX2P5AREIEIWHTXK7U22TEWDMRFQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:36.166039237 (72h14m36.037122272s) [noise=true , address=145.79.227.41:4085 , piece_id=ODJABK4R6SHNXNNQ47WI7W4YB4NR5A6QDTUE4WXVUFBVE7X3AJCQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:36.166480136 (72h14m36.037563181s) [noise=true , address=145.79.227.25:4619 , piece_id=CJJ7CONWO42FFSV7QMB4JSATW7BUDIS6KJXYX2SIBQMWE3UTYA4A, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:36.167642099 (72h14m36.038725134s) [noise=true , address=145.79.227.43:4157 , piece_id=3P7NOOCSNAUBKH4SVL2SUSQOADSA3IKRHTPRL4P7VGH62DL23LFQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:36.170345487 (72h14m36.041428532s) [noise=true , address=77.90.31.19:4842 , piece_id=D6RHPHJCJTV7I6UF5HQSPFFNVVHJDYXOIY3C747PLFFACW32INWA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:36.173595836 (72h14m36.044678881s) [noise=true , address=145.79.227.21:4475 , piece_id=PLY2YAALNNQKXXF5MZUUKDUV4HP7XUDRD3QXJTTKQI6FCGLHEA7A, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:36.185245680 (72h14m36.056328725s) [noise=true , address=145.79.227.28:4727 , piece_id=BSP57B4M3L7UIEDLVM2ASNQRK5BYLGTMAJ633QN4WFNS3OCNWDGQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:36.188436325 (72h14m36.05951937s) [noise=true , address=77.90.31.14:4661 , piece_id=BON7N46ZON2BYBD4UHNHPAECUF2OL7KV6B7Z2SZWVS6OGTEAK3RA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:36.199146960 (72h14m36.070229995s) [noise=true , address=145.79.227.38:4978 , piece_id=2AI2DCS6TNPJGURTBMNDKT5N3GISWDVOC5VTLIP6MVUAEWP2V3ZQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:36.484809459 (72h14m36.355892504s) [noise=true , address=145.79.227.39:4015 , piece_id=AUU7NTJQWR52XFWHVONDFHZHKXPOUS2W4UL4OF4XGTWV4ABETSHQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:37.179185330 (72h14m37.050268365s) [noise=true , address=145.79.227.22:4510 , piece_id=BVTVWVC7VRYL7TREJ63BRWY6LAHU2N5AHYJR3JRHQPPEEITTMFVQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:37.179203285 (72h14m37.05028633s) [seg_pos={0 0}, upload=63248]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:43:37.179369642 (72h14m37.050452687s) [noise=true , address=145.79.227.26:4653 , piece_id=LMBLPSOTMKPFS4KGAJOSRR2EM2MVKJIXSRE5MIB4SF552S2O3QYQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: expected piece hash; context canceled; EOF
Dec 8 15:43:37.179375573 (72h14m37.050458608s) [noise=true , address=145.79.227.26:4653 , piece_id=LMBLPSOTMKPFS4KGAJOSRR2EM2MVKJIXSRE5MIB4SF552S2O3QYQ, seg_pos={0 0}, upload=63248]: Long tail cancellation due to optimal threshold
Dec 8 15:43:37.179483609 (72h14m37.050566654s) [noise=true , address=145.79.227.29:4943 , piece_id=TJFW4JKEHHCDTMPVS46BI4OYF35BVHE3VIFM6LD2CJKRFCZXL5FA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF
Dec 8 15:43:37.179488698 (72h14m37.050571743s) [noise=true , address=145.79.227.29:4943 , piece_id=TJFW4JKEHHCDTMPVS46BI4OYF35BVHE3VIFM6LD2CJKRFCZXL5FA, seg_pos={0 0}, upload=63248]: Long tail cancellation due to optimal threshold
Dec 8 15:43:37.179679632 (72h14m37.050762687s) [noise=true , address=145.79.227.27:4689 , piece_id=LAMO54F6RV2YAJQNHCIOJNSAQGCXK23WHVJJ6VC2REYJ7LUW5R3Q, seg_pos={0 0}, upload=63248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ): protocol: write tcp 188.166.225.170:57686->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:57686->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57686->145.79.227.27:4689: use of closed network connection
Dec 8 15:43:37.179722844 (72h14m37.050805879s) [noise=true , address=145.79.227.27:4689 , piece_id=LAMO54F6RV2YAJQNHCIOJNSAQGCXK23WHVJJ6VC2REYJ7LUW5R3Q, seg_pos={0 0}, upload=63248]: Long tail cancellation due to optimal threshold
Dec 8 15:43:37.179754174 (72h14m37.050837209s) [seg_pos={0 0}, upload=63248]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:43:37.179977008 (72h14m37.051060043s) [upload=63248]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:43:37.179985915 (72h14m37.05106895s) [upload=63248]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:43:37.492140220 (72h14m37.363223265s) [upload=63248]: Done uploading segments...
Dec 8 15:43:37.492158695 (72h14m37.36324173s) [upload=63248]: Upload finished. err: <nil>
Duration: 7.250943941s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
6.647928776s trace:
https://jaeger.dev.storj.io/trace/153706e03a29fd88
FAIL (2025-12-08 15:44:25 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 6.647960727s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
6.169391285s trace:
https://jaeger.dev.storj.io/trace/4195524620a3e2c2
FAIL (2025-12-08 15:44:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment
Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments...
Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces.
Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments...
Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded
Duration: 6.169421272s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
31 out of 60 is failed (threshold=6) [XXXXXX.X.XXXXX.XXX..X....X...X...X.X.X.....XX.X.X.XXXX.X...X] duration:
5.001726504s trace:
https://jaeger.dev.storj.io/trace/35c49533edd380ce
FAIL (2025-12-08 15:44:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001754788s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.............................X.........X...X..............X.] duration:
11.766829916s trace:
https://jaeger.dev.storj.io/trace/3711a2689ed652fd
FAIL (2025-12-08 15:44:15 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: VKXVHWX7LFKBH4E32F6CMAHMQX2P5AREIEIWHTXK7U22TEWDMRFQ) context deadline exceeded; piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: AUU7NTJQWR52XFWHVONDFHZHKXPOUS2W4UL4OF4XGTWV4ABETSHQ) context deadline exceeded; piecestore: (Node ID: 12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, Piece ID: 2AI2DCS6TNPJGURTBMNDKT5N3GISWDVOC5VTLIP6MVUAEWP2V3ZQ) context deadline exceeded
======LOGS======
Dec 8 15:43:33.737263649 (72h14m33.608346694s) [upload=63248]: Starting upload
Dec 8 15:43:33.737410899 (72h14m33.608493944s) [upload=63248]: Done starting upload
Dec 8 15:43:33.737492094 (72h14m33.608575149s) [upload=63248]: Uploading segments...
Dec 8 15:43:33.739621138 (72h14m33.610704183s) [upload=63248]: Got next segment. Inline: false
Dec 8 15:43:33.739675270 (72h14m33.610758305s) [upload=63248]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:43:33.739681903 (72h14m33.610764938s) [upload=63248]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:43:33.988217234 (72h14m33.859300269s) [seg_pos={0 0}, upload=63248]: Begin upload segment...
Dec 8 15:43:33.988366548 (72h14m33.859449583s) [seg_pos={0 0}, upload=63248]: Done begin upload segment.
Dec 8 15:43:33.988395153 (72h14m33.859478198s) [noise=true , address=77.90.31.19:4842 , piece_id=D6RHPHJCJTV7I6UF5HQSPFFNVVHJDYXOIY3C747PLFFACW32INWA, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.988434698 (72h14m33.859517763s) [noise=true , address=145.79.227.27:4689 , piece_id=LAMO54F6RV2YAJQNHCIOJNSAQGCXK23WHVJJ6VC2REYJ7LUW5R3Q, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.989099694 (72h14m33.860182729s) [noise=true , address=145.79.227.29:4943 , piece_id=TJFW4JKEHHCDTMPVS46BI4OYF35BVHE3VIFM6LD2CJKRFCZXL5FA, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.989662838 (72h14m33.860745873s) [noise=true , address=145.79.227.22:4510 , piece_id=BVTVWVC7VRYL7TREJ63BRWY6LAHU2N5AHYJR3JRHQPPEEITTMFVQ, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.990143793 (72h14m33.861226828s) [noise=true , address=145.79.227.43:4157 , piece_id=3P7NOOCSNAUBKH4SVL2SUSQOADSA3IKRHTPRL4P7VGH62DL23LFQ, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.990589972 (72h14m33.861673007s) [noise=true , address=77.90.31.10:4519 , piece_id=PGLC7MAH7IJ7GP6DTN65WUESDI3Y2JNDRVFR3QJJBNZV2H2SFJSA, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.991038847 (72h14m33.862121882s) [noise=true , address=145.79.227.39:4015 , piece_id=AUU7NTJQWR52XFWHVONDFHZHKXPOUS2W4UL4OF4XGTWV4ABETSHQ, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.991412218 (72h14m33.862495263s) [noise=true , address=145.79.227.38:4978 , piece_id=2AI2DCS6TNPJGURTBMNDKT5N3GISWDVOC5VTLIP6MVUAEWP2V3ZQ, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.991728711 (72h14m33.862811746s) [noise=true , address=145.79.227.38:4979 , piece_id=4A4TZ2YT5KBQVYX7XSKIUMFJBLR3R7NJOZCVXEVVIK67VC744VXA, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.992474100 (72h14m33.863557145s) [noise=true , address=145.79.227.21:4475 , piece_id=PLY2YAALNNQKXXF5MZUUKDUV4HP7XUDRD3QXJTTKQI6FCGLHEA7A, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.992900823 (72h14m33.863983858s) [noise=true , address=145.79.227.28:4727 , piece_id=BSP57B4M3L7UIEDLVM2ASNQRK5BYLGTMAJ633QN4WFNS3OCNWDGQ, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.993301476 (72h14m33.864384521s) [noise=true , address=145.79.227.41:4085 , piece_id=ODJABK4R6SHNXNNQ47WI7W4YB4NR5A6QDTUE4WXVUFBVE7X3AJCQ, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.993926856 (72h14m33.865009922s) [noise=true , address=145.79.227.25:4619 , piece_id=CJJ7CONWO42FFSV7QMB4JSATW7BUDIS6KJXYX2SIBQMWE3UTYA4A, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.994260803 (72h14m33.865343838s) [noise=true , address=77.90.31.14:4661 , piece_id=BON7N46ZON2BYBD4UHNHPAECUF2OL7KV6B7Z2SZWVS6OGTEAK3RA, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.994756687 (72h14m33.865839732s) [noise=true , address=145.79.227.31:4799 , piece_id=VKXVHWX7LFKBH4E32F6CMAHMQX2P5AREIEIWHTXK7U22TEWDMRFQ, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.995123304 (72h14m33.866206349s) [noise=true , address=145.79.227.26:4653 , piece_id=LMBLPSOTMKPFS4KGAJOSRR2EM2MVKJIXSRE5MIB4SF552S2O3QYQ, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:35.816987512 (72h14m35.688070547s) [upload=63248]: Next returned nil segment
Dec 8 15:43:35.817000286 (72h14m35.688083331s) [upload=63248]: Waiting for error group managing segments...
Dec 8 15:43:36.138037836 (72h14m36.009120871s) [noise=true , address=77.90.31.10:4519 , piece_id=PGLC7MAH7IJ7GP6DTN65WUESDI3Y2JNDRVFR3QJJBNZV2H2SFJSA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:36.161877783 (72h14m36.032960818s) [noise=true , address=145.79.227.38:4979 , piece_id=4A4TZ2YT5KBQVYX7XSKIUMFJBLR3R7NJOZCVXEVVIK67VC744VXA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:36.164129480 (72h14m36.035212525s) [noise=true , address=145.79.227.31:4799 , piece_id=VKXVHWX7LFKBH4E32F6CMAHMQX2P5AREIEIWHTXK7U22TEWDMRFQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:36.166039237 (72h14m36.037122272s) [noise=true , address=145.79.227.41:4085 , piece_id=ODJABK4R6SHNXNNQ47WI7W4YB4NR5A6QDTUE4WXVUFBVE7X3AJCQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:36.166480136 (72h14m36.037563181s) [noise=true , address=145.79.227.25:4619 , piece_id=CJJ7CONWO42FFSV7QMB4JSATW7BUDIS6KJXYX2SIBQMWE3UTYA4A, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:36.167642099 (72h14m36.038725134s) [noise=true , address=145.79.227.43:4157 , piece_id=3P7NOOCSNAUBKH4SVL2SUSQOADSA3IKRHTPRL4P7VGH62DL23LFQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:36.170345487 (72h14m36.041428532s) [noise=true , address=77.90.31.19:4842 , piece_id=D6RHPHJCJTV7I6UF5HQSPFFNVVHJDYXOIY3C747PLFFACW32INWA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:36.173595836 (72h14m36.044678881s) [noise=true , address=145.79.227.21:4475 , piece_id=PLY2YAALNNQKXXF5MZUUKDUV4HP7XUDRD3QXJTTKQI6FCGLHEA7A, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:36.185245680 (72h14m36.056328725s) [noise=true , address=145.79.227.28:4727 , piece_id=BSP57B4M3L7UIEDLVM2ASNQRK5BYLGTMAJ633QN4WFNS3OCNWDGQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:36.188436325 (72h14m36.05951937s) [noise=true , address=77.90.31.14:4661 , piece_id=BON7N46ZON2BYBD4UHNHPAECUF2OL7KV6B7Z2SZWVS6OGTEAK3RA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:36.199146960 (72h14m36.070229995s) [noise=true , address=145.79.227.38:4978 , piece_id=2AI2DCS6TNPJGURTBMNDKT5N3GISWDVOC5VTLIP6MVUAEWP2V3ZQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:36.484809459 (72h14m36.355892504s) [noise=true , address=145.79.227.39:4015 , piece_id=AUU7NTJQWR52XFWHVONDFHZHKXPOUS2W4UL4OF4XGTWV4ABETSHQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:37.179185330 (72h14m37.050268365s) [noise=true , address=145.79.227.22:4510 , piece_id=BVTVWVC7VRYL7TREJ63BRWY6LAHU2N5AHYJR3JRHQPPEEITTMFVQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:37.179203285 (72h14m37.05028633s) [seg_pos={0 0}, upload=63248]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:43:37.179369642 (72h14m37.050452687s) [noise=true , address=145.79.227.26:4653 , piece_id=LMBLPSOTMKPFS4KGAJOSRR2EM2MVKJIXSRE5MIB4SF552S2O3QYQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: expected piece hash; context canceled; EOF
Dec 8 15:43:37.179375573 (72h14m37.050458608s) [noise=true , address=145.79.227.26:4653 , piece_id=LMBLPSOTMKPFS4KGAJOSRR2EM2MVKJIXSRE5MIB4SF552S2O3QYQ, seg_pos={0 0}, upload=63248]: Long tail cancellation due to optimal threshold
Dec 8 15:43:37.179483609 (72h14m37.050566654s) [noise=true , address=145.79.227.29:4943 , piece_id=TJFW4JKEHHCDTMPVS46BI4OYF35BVHE3VIFM6LD2CJKRFCZXL5FA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF
Dec 8 15:43:37.179488698 (72h14m37.050571743s) [noise=true , address=145.79.227.29:4943 , piece_id=TJFW4JKEHHCDTMPVS46BI4OYF35BVHE3VIFM6LD2CJKRFCZXL5FA, seg_pos={0 0}, upload=63248]: Long tail cancellation due to optimal threshold
Dec 8 15:43:37.179679632 (72h14m37.050762687s) [noise=true , address=145.79.227.27:4689 , piece_id=LAMO54F6RV2YAJQNHCIOJNSAQGCXK23WHVJJ6VC2REYJ7LUW5R3Q, seg_pos={0 0}, upload=63248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ): protocol: write tcp 188.166.225.170:57686->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:57686->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57686->145.79.227.27:4689: use of closed network connection
Dec 8 15:43:37.179722844 (72h14m37.050805879s) [noise=true , address=145.79.227.27:4689 , piece_id=LAMO54F6RV2YAJQNHCIOJNSAQGCXK23WHVJJ6VC2REYJ7LUW5R3Q, seg_pos={0 0}, upload=63248]: Long tail cancellation due to optimal threshold
Dec 8 15:43:37.179754174 (72h14m37.050837209s) [seg_pos={0 0}, upload=63248]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:43:37.179977008 (72h14m37.051060043s) [upload=63248]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:43:37.179985915 (72h14m37.05106895s) [upload=63248]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:43:37.492140220 (72h14m37.363223265s) [upload=63248]: Done uploading segments...
Dec 8 15:43:37.492158695 (72h14m37.36324173s) [upload=63248]: Upload finished. err: <nil>
Duration: 11.766844994s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [..........X............X.X.XX.XX.XX..X.....XX...X.X.X.X....X] duration:
30.001183227s trace:
https://jaeger.dev.storj.io/trace/580a9232c1e4b665
FAIL (2025-12-08 15:44:15 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001195019s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..............................X.........X...X..............X] duration:
30.002806554s trace:
https://jaeger.dev.storj.io/trace/3711a2689ed652fd
FAIL (2025-12-08 15:44:03 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: VKXVHWX7LFKBH4E32F6CMAHMQX2P5AREIEIWHTXK7U22TEWDMRFQ) context deadline exceeded; piecestore: (Node ID: 18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, Piece ID: AUU7NTJQWR52XFWHVONDFHZHKXPOUS2W4UL4OF4XGTWV4ABETSHQ) context deadline exceeded; piecestore: (Node ID: 12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, Piece ID: 2AI2DCS6TNPJGURTBMNDKT5N3GISWDVOC5VTLIP6MVUAEWP2V3ZQ) context deadline exceeded
======LOGS======
Dec 8 15:43:33.737263649 (72h14m33.608346694s) [upload=63248]: Starting upload
Dec 8 15:43:33.737410899 (72h14m33.608493944s) [upload=63248]: Done starting upload
Dec 8 15:43:33.737492094 (72h14m33.608575149s) [upload=63248]: Uploading segments...
Dec 8 15:43:33.739621138 (72h14m33.610704183s) [upload=63248]: Got next segment. Inline: false
Dec 8 15:43:33.739675270 (72h14m33.610758305s) [upload=63248]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:43:33.739681903 (72h14m33.610764938s) [upload=63248]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:43:33.988217234 (72h14m33.859300269s) [seg_pos={0 0}, upload=63248]: Begin upload segment...
Dec 8 15:43:33.988366548 (72h14m33.859449583s) [seg_pos={0 0}, upload=63248]: Done begin upload segment.
Dec 8 15:43:33.988395153 (72h14m33.859478198s) [noise=true , address=77.90.31.19:4842 , piece_id=D6RHPHJCJTV7I6UF5HQSPFFNVVHJDYXOIY3C747PLFFACW32INWA, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.988434698 (72h14m33.859517763s) [noise=true , address=145.79.227.27:4689 , piece_id=LAMO54F6RV2YAJQNHCIOJNSAQGCXK23WHVJJ6VC2REYJ7LUW5R3Q, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.989099694 (72h14m33.860182729s) [noise=true , address=145.79.227.29:4943 , piece_id=TJFW4JKEHHCDTMPVS46BI4OYF35BVHE3VIFM6LD2CJKRFCZXL5FA, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.989662838 (72h14m33.860745873s) [noise=true , address=145.79.227.22:4510 , piece_id=BVTVWVC7VRYL7TREJ63BRWY6LAHU2N5AHYJR3JRHQPPEEITTMFVQ, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.990143793 (72h14m33.861226828s) [noise=true , address=145.79.227.43:4157 , piece_id=3P7NOOCSNAUBKH4SVL2SUSQOADSA3IKRHTPRL4P7VGH62DL23LFQ, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.990589972 (72h14m33.861673007s) [noise=true , address=77.90.31.10:4519 , piece_id=PGLC7MAH7IJ7GP6DTN65WUESDI3Y2JNDRVFR3QJJBNZV2H2SFJSA, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.991038847 (72h14m33.862121882s) [noise=true , address=145.79.227.39:4015 , piece_id=AUU7NTJQWR52XFWHVONDFHZHKXPOUS2W4UL4OF4XGTWV4ABETSHQ, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.991412218 (72h14m33.862495263s) [noise=true , address=145.79.227.38:4978 , piece_id=2AI2DCS6TNPJGURTBMNDKT5N3GISWDVOC5VTLIP6MVUAEWP2V3ZQ, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.991728711 (72h14m33.862811746s) [noise=true , address=145.79.227.38:4979 , piece_id=4A4TZ2YT5KBQVYX7XSKIUMFJBLR3R7NJOZCVXEVVIK67VC744VXA, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.992474100 (72h14m33.863557145s) [noise=true , address=145.79.227.21:4475 , piece_id=PLY2YAALNNQKXXF5MZUUKDUV4HP7XUDRD3QXJTTKQI6FCGLHEA7A, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.992900823 (72h14m33.863983858s) [noise=true , address=145.79.227.28:4727 , piece_id=BSP57B4M3L7UIEDLVM2ASNQRK5BYLGTMAJ633QN4WFNS3OCNWDGQ, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.993301476 (72h14m33.864384521s) [noise=true , address=145.79.227.41:4085 , piece_id=ODJABK4R6SHNXNNQ47WI7W4YB4NR5A6QDTUE4WXVUFBVE7X3AJCQ, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.993926856 (72h14m33.865009922s) [noise=true , address=145.79.227.25:4619 , piece_id=CJJ7CONWO42FFSV7QMB4JSATW7BUDIS6KJXYX2SIBQMWE3UTYA4A, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.994260803 (72h14m33.865343838s) [noise=true , address=77.90.31.14:4661 , piece_id=BON7N46ZON2BYBD4UHNHPAECUF2OL7KV6B7Z2SZWVS6OGTEAK3RA, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.994756687 (72h14m33.865839732s) [noise=true , address=145.79.227.31:4799 , piece_id=VKXVHWX7LFKBH4E32F6CMAHMQX2P5AREIEIWHTXK7U22TEWDMRFQ, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:33.995123304 (72h14m33.866206349s) [noise=true , address=145.79.227.26:4653 , piece_id=LMBLPSOTMKPFS4KGAJOSRR2EM2MVKJIXSRE5MIB4SF552S2O3QYQ, seg_pos={0 0}, upload=63248]: Uploading piece...
Dec 8 15:43:35.816987512 (72h14m35.688070547s) [upload=63248]: Next returned nil segment
Dec 8 15:43:35.817000286 (72h14m35.688083331s) [upload=63248]: Waiting for error group managing segments...
Dec 8 15:43:36.138037836 (72h14m36.009120871s) [noise=true , address=77.90.31.10:4519 , piece_id=PGLC7MAH7IJ7GP6DTN65WUESDI3Y2JNDRVFR3QJJBNZV2H2SFJSA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:36.161877783 (72h14m36.032960818s) [noise=true , address=145.79.227.38:4979 , piece_id=4A4TZ2YT5KBQVYX7XSKIUMFJBLR3R7NJOZCVXEVVIK67VC744VXA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:36.164129480 (72h14m36.035212525s) [noise=true , address=145.79.227.31:4799 , piece_id=VKXVHWX7LFKBH4E32F6CMAHMQX2P5AREIEIWHTXK7U22TEWDMRFQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:36.166039237 (72h14m36.037122272s) [noise=true , address=145.79.227.41:4085 , piece_id=ODJABK4R6SHNXNNQ47WI7W4YB4NR5A6QDTUE4WXVUFBVE7X3AJCQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:36.166480136 (72h14m36.037563181s) [noise=true , address=145.79.227.25:4619 , piece_id=CJJ7CONWO42FFSV7QMB4JSATW7BUDIS6KJXYX2SIBQMWE3UTYA4A, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:36.167642099 (72h14m36.038725134s) [noise=true , address=145.79.227.43:4157 , piece_id=3P7NOOCSNAUBKH4SVL2SUSQOADSA3IKRHTPRL4P7VGH62DL23LFQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:36.170345487 (72h14m36.041428532s) [noise=true , address=77.90.31.19:4842 , piece_id=D6RHPHJCJTV7I6UF5HQSPFFNVVHJDYXOIY3C747PLFFACW32INWA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:36.173595836 (72h14m36.044678881s) [noise=true , address=145.79.227.21:4475 , piece_id=PLY2YAALNNQKXXF5MZUUKDUV4HP7XUDRD3QXJTTKQI6FCGLHEA7A, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:36.185245680 (72h14m36.056328725s) [noise=true , address=145.79.227.28:4727 , piece_id=BSP57B4M3L7UIEDLVM2ASNQRK5BYLGTMAJ633QN4WFNS3OCNWDGQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:36.188436325 (72h14m36.05951937s) [noise=true , address=77.90.31.14:4661 , piece_id=BON7N46ZON2BYBD4UHNHPAECUF2OL7KV6B7Z2SZWVS6OGTEAK3RA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:36.199146960 (72h14m36.070229995s) [noise=true , address=145.79.227.38:4978 , piece_id=2AI2DCS6TNPJGURTBMNDKT5N3GISWDVOC5VTLIP6MVUAEWP2V3ZQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:36.484809459 (72h14m36.355892504s) [noise=true , address=145.79.227.39:4015 , piece_id=AUU7NTJQWR52XFWHVONDFHZHKXPOUS2W4UL4OF4XGTWV4ABETSHQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:37.179185330 (72h14m37.050268365s) [noise=true , address=145.79.227.22:4510 , piece_id=BVTVWVC7VRYL7TREJ63BRWY6LAHU2N5AHYJR3JRHQPPEEITTMFVQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: <nil>
Dec 8 15:43:37.179203285 (72h14m37.05028633s) [seg_pos={0 0}, upload=63248]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:43:37.179369642 (72h14m37.050452687s) [noise=true , address=145.79.227.26:4653 , piece_id=LMBLPSOTMKPFS4KGAJOSRR2EM2MVKJIXSRE5MIB4SF552S2O3QYQ, seg_pos={0 0}, upload=63248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CJCyYubAEcM8Tf9daXGiMaazEQRczgPWF8G2Xmox3SE9AWueQ): protocol: expected piece hash; context canceled; EOF
Dec 8 15:43:37.179375573 (72h14m37.050458608s) [noise=true , address=145.79.227.26:4653 , piece_id=LMBLPSOTMKPFS4KGAJOSRR2EM2MVKJIXSRE5MIB4SF552S2O3QYQ, seg_pos={0 0}, upload=63248]: Long tail cancellation due to optimal threshold
Dec 8 15:43:37.179483609 (72h14m37.050566654s) [noise=true , address=145.79.227.29:4943 , piece_id=TJFW4JKEHHCDTMPVS46BI4OYF35BVHE3VIFM6LD2CJKRFCZXL5FA, seg_pos={0 0}, upload=63248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF
Dec 8 15:43:37.179488698 (72h14m37.050571743s) [noise=true , address=145.79.227.29:4943 , piece_id=TJFW4JKEHHCDTMPVS46BI4OYF35BVHE3VIFM6LD2CJKRFCZXL5FA, seg_pos={0 0}, upload=63248]: Long tail cancellation due to optimal threshold
Dec 8 15:43:37.179679632 (72h14m37.050762687s) [noise=true , address=145.79.227.27:4689 , piece_id=LAMO54F6RV2YAJQNHCIOJNSAQGCXK23WHVJJ6VC2REYJ7LUW5R3Q, seg_pos={0 0}, upload=63248]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QXoPNv44qGgVJZR9oRSmogH4paPpS4HX5jLBJdef1X9ev4stQ): protocol: write tcp 188.166.225.170:57686->145.79.227.27:4689: use of closed network connection; write tcp 188.166.225.170:57686->145.79.227.27:4689: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57686->145.79.227.27:4689: use of closed network connection
Dec 8 15:43:37.179722844 (72h14m37.050805879s) [noise=true , address=145.79.227.27:4689 , piece_id=LAMO54F6RV2YAJQNHCIOJNSAQGCXK23WHVJJ6VC2REYJ7LUW5R3Q, seg_pos={0 0}, upload=63248]: Long tail cancellation due to optimal threshold
Dec 8 15:43:37.179754174 (72h14m37.050837209s) [seg_pos={0 0}, upload=63248]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:43:37.179977008 (72h14m37.051060043s) [upload=63248]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:43:37.179985915 (72h14m37.05106895s) [upload=63248]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:43:37.492140220 (72h14m37.363223265s) [upload=63248]: Done uploading segments...
Dec 8 15:43:37.492158695 (72h14m37.36324173s) [upload=63248]: Upload finished. err: <nil>
Duration: 30.00283058s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
6.107221483s trace:
https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182
FAIL (2025-12-08 15:43:49 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 6.107247071s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [...........X............X.X.XX.XX.XX..X.....XX...X.X.X.X....] duration:
27.526750424s trace:
https://jaeger.dev.storj.io/trace/7b0090eb8f29bbc6
FAIL (2025-12-08 15:43:45 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 27.526999609s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...........................X.........................X..X...] duration:
5.722417633s trace:
https://jaeger.dev.storj.io/trace/43c87fd861da304
FAIL (2025-12-08 15:43:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.722525268s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
31 out of 60 is failed (threshold=6) [XXXXXXX.X.XXXXX.XXX..X....X...X...X.X.X.....XX.X.X.XXXX.X...] duration:
3.435860338s trace:
https://jaeger.dev.storj.io/trace/52dbb0834262bbe5
FAIL (2025-12-08 15:43:43 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.435885187s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
4.975490976s trace:
https://jaeger.dev.storj.io/trace/4195524620a3e2c2
FAIL (2025-12-08 15:43:31 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment
Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments...
Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces.
Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments...
Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded
Duration: 4.975514291s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
6.037724823s trace:
https://jaeger.dev.storj.io/trace/153706e03a29fd88
FAIL (2025-12-08 15:43:16 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 6.037756744s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
5.98177279s trace:
https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182
FAIL (2025-12-08 15:43:12 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.981804672s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...............................X.........X...X..............] duration:
9.753961747s trace:
https://jaeger.dev.storj.io/trace/631c2b6153f16a63
FAIL (2025-12-08 15:43:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA) context deadline exceeded; piecestore: (Node ID: 13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, Piece ID: 5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A) context deadline exceeded; piecestore: (Node ID: 12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, Piece ID: DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA) context deadline exceeded
======LOGS======
Dec 8 15:31:47.733980955 (72h2m47.605064s) [upload=63079]: Starting upload
Dec 8 15:31:47.734113437 (72h2m47.605196482s) [upload=63079]: Done starting upload
Dec 8 15:31:47.734220060 (72h2m47.605303095s) [upload=63079]: Uploading segments...
Dec 8 15:31:47.734376698 (72h2m47.605459743s) [upload=63079]: Got next segment. Inline: false
Dec 8 15:31:47.734425131 (72h2m47.605508166s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:31:47.734431473 (72h2m47.605514518s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:31:48.240451445 (72h2m48.11153449s) [seg_pos={0 0}, upload=63079]: Begin upload segment...
Dec 8 15:31:48.240553519 (72h2m48.111636554s) [seg_pos={0 0}, upload=63079]: Done begin upload segment.
Dec 8 15:31:48.240584428 (72h2m48.111667473s) [noise=true , address=145.79.227.27:4691 , piece_id=TSGC3VYZETJX3AR7AFMC3WV5CIJAKQLVX6W2CA7PIZYYAOEZXYOA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.241909641 (72h2m48.112992686s) [noise=true , address=77.90.31.19:4841 , piece_id=QFLCO4T2GILZQ7QLAMQVMJMPNNPVXYX72XMFN2G3WLJ5MCSERAQQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.242622299 (72h2m48.113705334s) [noise=true , address=145.79.227.2:4113 , piece_id=VBI4ITMJCYEKRNQ2R5ZVL4PJYKYSMAGZDN7LJGXBT5DT5MKCS2EQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.243034072 (72h2m48.114117107s) [noise=true , address=145.79.227.31:4798 , piece_id=VIL36YB442SIWR6UHFDY4F62HHDJS2IB5T6YDOOAGMJIJEJNHZAQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.243641960 (72h2m48.114725005s) [noise=true , address=145.79.227.24:4581 , piece_id=JK2YXXKH2D33KLTYJ2JNW4I6HKQGJWR3BAUULSMUFKL4A2DHQZVA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244059034 (72h2m48.115142069s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244487580 (72h2m48.115570615s) [noise=true , address=145.79.227.20:4438 , piece_id=ZVQYBOT6NDTUNJFHWBBXBFALMQFFNZPHTVA2NIPJ2LGEOFLLOLEA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244706898 (72h2m48.115789944s) [noise=true , address=145.79.227.16:4910 , piece_id=JTWXUSJITQMTHBBRTRTSAYXNY3YN7H5WFAUTWAJEREMNMTDAB4RQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.245225194 (72h2m48.11630823s) [noise=true , address=145.79.227.41:4085 , piece_id=Z4NSLWRCQD4D6YHT7M27BGO3CLWK5WOFZ377G44SRUK4FJBA7GXA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.245653140 (72h2m48.116736185s) [noise=true , address=145.79.227.38:4977 , piece_id=Y626H42FV2KSMBMRZHC2S4R4DJLZA6I3XFEMMAOS62FTVRRZSS5A, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246124207 (72h2m48.117207242s) [noise=true , address=145.79.227.25:4618 , piece_id=ALUOULKKNPL56O2PATZO5ZQYSY6JLRAJ23Y4ZNGJBHCS4U3ELVWA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246531512 (72h2m48.117614547s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246921264 (72h2m48.118004299s) [noise=true , address=145.79.227.28:4726 , piece_id=GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.247439812 (72h2m48.118522847s) [noise=true , address=145.79.227.16:4911 , piece_id=5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.249604864 (72h2m48.120687909s) [noise=true , address=145.79.227.34:4905 , piece_id=EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.250349092 (72h2m48.121432137s) [noise=true , address=145.79.227.44:4195 , piece_id=DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:49.352808394 (72h2m49.223891439s) [noise=true , address=77.90.31.19:4841 , piece_id=QFLCO4T2GILZQ7QLAMQVMJMPNNPVXYX72XMFN2G3WLJ5MCSERAQQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 8 15:31:50.097967294 (72h2m49.969050339s) [upload=63079]: Next returned nil segment
Dec 8 15:31:50.097984627 (72h2m49.969067683s) [upload=63079]: Waiting for error group managing segments...
Dec 8 15:31:50.416319661 (72h2m50.287402706s) [noise=true , address=145.79.227.31:4798 , piece_id=VIL36YB442SIWR6UHFDY4F62HHDJS2IB5T6YDOOAGMJIJEJNHZAQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.427033041 (72h2m50.298116096s) [noise=true , address=145.79.227.16:4910 , piece_id=JTWXUSJITQMTHBBRTRTSAYXNY3YN7H5WFAUTWAJEREMNMTDAB4RQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.430562030 (72h2m50.301645115s) [noise=true , address=145.79.227.24:4581 , piece_id=JK2YXXKH2D33KLTYJ2JNW4I6HKQGJWR3BAUULSMUFKL4A2DHQZVA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.466400595 (72h2m50.33748364s) [noise=true , address=145.79.227.34:4905 , piece_id=EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.472873420 (72h2m50.343956455s) [noise=true , address=145.79.227.27:4691 , piece_id=TSGC3VYZETJX3AR7AFMC3WV5CIJAKQLVX6W2CA7PIZYYAOEZXYOA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.531272483 (72h2m50.402355528s) [noise=true , address=145.79.227.2:4113 , piece_id=VBI4ITMJCYEKRNQ2R5ZVL4PJYKYSMAGZDN7LJGXBT5DT5MKCS2EQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.729877357 (72h2m50.600960402s) [noise=true , address=145.79.227.16:4911 , piece_id=5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.779057345 (72h2m50.65014039s) [noise=true , address=145.79.227.25:4618 , piece_id=ALUOULKKNPL56O2PATZO5ZQYSY6JLRAJ23Y4ZNGJBHCS4U3ELVWA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.104680797 (72h2m50.975763832s) [noise=true , address=145.79.227.28:4726 , piece_id=GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.116683803 (72h2m50.987766848s) [noise=true , address=145.79.227.38:4977 , piece_id=Y626H42FV2KSMBMRZHC2S4R4DJLZA6I3XFEMMAOS62FTVRRZSS5A, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.132265363 (72h2m51.003348398s) [noise=true , address=145.79.227.20:4438 , piece_id=ZVQYBOT6NDTUNJFHWBBXBFALMQFFNZPHTVA2NIPJ2LGEOFLLOLEA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.146914038 (72h2m51.017997083s) [noise=true , address=145.79.227.44:4195 , piece_id=DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.452416276 (72h2m51.323499321s) [noise=true , address=145.79.227.41:4085 , piece_id=Z4NSLWRCQD4D6YHT7M27BGO3CLWK5WOFZ377G44SRUK4FJBA7GXA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.452447905 (72h2m51.32353095s) [seg_pos={0 0}, upload=63079]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:31:51.452802852 (72h2m51.323885897s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF
Dec 8 15:31:51.452816517 (72h2m51.323899562s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Long tail cancellation due to optimal threshold
Dec 8 15:31:51.452931045 (72h2m51.32401409s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: read tcp 188.166.225.170:41450->145.79.227.39:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:31:51.452966172 (72h2m51.324049217s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Long tail cancellation due to optimal threshold
Dec 8 15:31:51.452996309 (72h2m51.324079354s) [seg_pos={0 0}, upload=63079]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:31:51.453109756 (72h2m51.324192801s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:31:51.453116439 (72h2m51.324199484s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:31:52.077764883 (72h2m51.948847938s) [upload=63079]: Done uploading segments...
Dec 8 15:31:52.077787646 (72h2m51.948870691s) [upload=63079]: Upload finished. err: <nil>
Duration: 9.753993047s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
32 out of 60 is failed (threshold=6) [XXXXXXXX.X.XXXXX.XXX..X....X...X...X.X.X.....XX.X.X.XXXX.X..] duration:
3.472909199s trace:
https://jaeger.dev.storj.io/trace/52dbb0834262bbe5
FAIL (2025-12-08 15:43:04 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.472942994s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
4.805499689s trace:
https://jaeger.dev.storj.io/trace/4195524620a3e2c2
FAIL (2025-12-08 15:43:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment
Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments...
Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces.
Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments...
Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded
Duration: 4.805531941s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [................................X.........X...X.............] duration:
9.414411867s trace:
https://jaeger.dev.storj.io/trace/631c2b6153f16a63
FAIL (2025-12-08 15:42:37 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA) context deadline exceeded; piecestore: (Node ID: 13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, Piece ID: 5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A) context deadline exceeded; piecestore: (Node ID: 12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, Piece ID: DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA) context deadline exceeded
======LOGS======
Dec 8 15:31:47.733980955 (72h2m47.605064s) [upload=63079]: Starting upload
Dec 8 15:31:47.734113437 (72h2m47.605196482s) [upload=63079]: Done starting upload
Dec 8 15:31:47.734220060 (72h2m47.605303095s) [upload=63079]: Uploading segments...
Dec 8 15:31:47.734376698 (72h2m47.605459743s) [upload=63079]: Got next segment. Inline: false
Dec 8 15:31:47.734425131 (72h2m47.605508166s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:31:47.734431473 (72h2m47.605514518s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:31:48.240451445 (72h2m48.11153449s) [seg_pos={0 0}, upload=63079]: Begin upload segment...
Dec 8 15:31:48.240553519 (72h2m48.111636554s) [seg_pos={0 0}, upload=63079]: Done begin upload segment.
Dec 8 15:31:48.240584428 (72h2m48.111667473s) [noise=true , address=145.79.227.27:4691 , piece_id=TSGC3VYZETJX3AR7AFMC3WV5CIJAKQLVX6W2CA7PIZYYAOEZXYOA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.241909641 (72h2m48.112992686s) [noise=true , address=77.90.31.19:4841 , piece_id=QFLCO4T2GILZQ7QLAMQVMJMPNNPVXYX72XMFN2G3WLJ5MCSERAQQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.242622299 (72h2m48.113705334s) [noise=true , address=145.79.227.2:4113 , piece_id=VBI4ITMJCYEKRNQ2R5ZVL4PJYKYSMAGZDN7LJGXBT5DT5MKCS2EQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.243034072 (72h2m48.114117107s) [noise=true , address=145.79.227.31:4798 , piece_id=VIL36YB442SIWR6UHFDY4F62HHDJS2IB5T6YDOOAGMJIJEJNHZAQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.243641960 (72h2m48.114725005s) [noise=true , address=145.79.227.24:4581 , piece_id=JK2YXXKH2D33KLTYJ2JNW4I6HKQGJWR3BAUULSMUFKL4A2DHQZVA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244059034 (72h2m48.115142069s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244487580 (72h2m48.115570615s) [noise=true , address=145.79.227.20:4438 , piece_id=ZVQYBOT6NDTUNJFHWBBXBFALMQFFNZPHTVA2NIPJ2LGEOFLLOLEA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244706898 (72h2m48.115789944s) [noise=true , address=145.79.227.16:4910 , piece_id=JTWXUSJITQMTHBBRTRTSAYXNY3YN7H5WFAUTWAJEREMNMTDAB4RQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.245225194 (72h2m48.11630823s) [noise=true , address=145.79.227.41:4085 , piece_id=Z4NSLWRCQD4D6YHT7M27BGO3CLWK5WOFZ377G44SRUK4FJBA7GXA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.245653140 (72h2m48.116736185s) [noise=true , address=145.79.227.38:4977 , piece_id=Y626H42FV2KSMBMRZHC2S4R4DJLZA6I3XFEMMAOS62FTVRRZSS5A, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246124207 (72h2m48.117207242s) [noise=true , address=145.79.227.25:4618 , piece_id=ALUOULKKNPL56O2PATZO5ZQYSY6JLRAJ23Y4ZNGJBHCS4U3ELVWA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246531512 (72h2m48.117614547s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246921264 (72h2m48.118004299s) [noise=true , address=145.79.227.28:4726 , piece_id=GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.247439812 (72h2m48.118522847s) [noise=true , address=145.79.227.16:4911 , piece_id=5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.249604864 (72h2m48.120687909s) [noise=true , address=145.79.227.34:4905 , piece_id=EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.250349092 (72h2m48.121432137s) [noise=true , address=145.79.227.44:4195 , piece_id=DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:49.352808394 (72h2m49.223891439s) [noise=true , address=77.90.31.19:4841 , piece_id=QFLCO4T2GILZQ7QLAMQVMJMPNNPVXYX72XMFN2G3WLJ5MCSERAQQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 8 15:31:50.097967294 (72h2m49.969050339s) [upload=63079]: Next returned nil segment
Dec 8 15:31:50.097984627 (72h2m49.969067683s) [upload=63079]: Waiting for error group managing segments...
Dec 8 15:31:50.416319661 (72h2m50.287402706s) [noise=true , address=145.79.227.31:4798 , piece_id=VIL36YB442SIWR6UHFDY4F62HHDJS2IB5T6YDOOAGMJIJEJNHZAQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.427033041 (72h2m50.298116096s) [noise=true , address=145.79.227.16:4910 , piece_id=JTWXUSJITQMTHBBRTRTSAYXNY3YN7H5WFAUTWAJEREMNMTDAB4RQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.430562030 (72h2m50.301645115s) [noise=true , address=145.79.227.24:4581 , piece_id=JK2YXXKH2D33KLTYJ2JNW4I6HKQGJWR3BAUULSMUFKL4A2DHQZVA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.466400595 (72h2m50.33748364s) [noise=true , address=145.79.227.34:4905 , piece_id=EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.472873420 (72h2m50.343956455s) [noise=true , address=145.79.227.27:4691 , piece_id=TSGC3VYZETJX3AR7AFMC3WV5CIJAKQLVX6W2CA7PIZYYAOEZXYOA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.531272483 (72h2m50.402355528s) [noise=true , address=145.79.227.2:4113 , piece_id=VBI4ITMJCYEKRNQ2R5ZVL4PJYKYSMAGZDN7LJGXBT5DT5MKCS2EQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.729877357 (72h2m50.600960402s) [noise=true , address=145.79.227.16:4911 , piece_id=5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.779057345 (72h2m50.65014039s) [noise=true , address=145.79.227.25:4618 , piece_id=ALUOULKKNPL56O2PATZO5ZQYSY6JLRAJ23Y4ZNGJBHCS4U3ELVWA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.104680797 (72h2m50.975763832s) [noise=true , address=145.79.227.28:4726 , piece_id=GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.116683803 (72h2m50.987766848s) [noise=true , address=145.79.227.38:4977 , piece_id=Y626H42FV2KSMBMRZHC2S4R4DJLZA6I3XFEMMAOS62FTVRRZSS5A, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.132265363 (72h2m51.003348398s) [noise=true , address=145.79.227.20:4438 , piece_id=ZVQYBOT6NDTUNJFHWBBXBFALMQFFNZPHTVA2NIPJ2LGEOFLLOLEA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.146914038 (72h2m51.017997083s) [noise=true , address=145.79.227.44:4195 , piece_id=DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.452416276 (72h2m51.323499321s) [noise=true , address=145.79.227.41:4085 , piece_id=Z4NSLWRCQD4D6YHT7M27BGO3CLWK5WOFZ377G44SRUK4FJBA7GXA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.452447905 (72h2m51.32353095s) [seg_pos={0 0}, upload=63079]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:31:51.452802852 (72h2m51.323885897s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF
Dec 8 15:31:51.452816517 (72h2m51.323899562s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Long tail cancellation due to optimal threshold
Dec 8 15:31:51.452931045 (72h2m51.32401409s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: read tcp 188.166.225.170:41450->145.79.227.39:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:31:51.452966172 (72h2m51.324049217s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Long tail cancellation due to optimal threshold
Dec 8 15:31:51.452996309 (72h2m51.324079354s) [seg_pos={0 0}, upload=63079]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:31:51.453109756 (72h2m51.324192801s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:31:51.453116439 (72h2m51.324199484s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:31:52.077764883 (72h2m51.948847938s) [upload=63079]: Done uploading segments...
Dec 8 15:31:52.077787646 (72h2m51.948870691s) [upload=63079]: Upload finished. err: <nil>
Duration: 9.414531133s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [............X............X.X.XX.XX.XX..X.....XX...X.X.X.X...] duration:
5.937380067s trace:
https://jaeger.dev.storj.io/trace/7b0090eb8f29bbc6
FAIL (2025-12-08 15:42:37 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.937415766s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [............................X.........................X..X..] duration:
8.028472809s trace:
https://jaeger.dev.storj.io/trace/43c87fd861da304
FAIL (2025-12-08 15:42:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 8.028502905s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
7.446967375s trace:
https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182
FAIL (2025-12-08 15:42:31 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 7.446996363s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
5.064372697s trace:
https://jaeger.dev.storj.io/trace/153706e03a29fd88
FAIL (2025-12-08 15:42:29 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 5.064385321s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
4.929052161s trace:
https://jaeger.dev.storj.io/trace/153706e03a29fd88
FAIL (2025-12-08 15:42:11 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 4.929106254s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
8.707507633s trace:
https://jaeger.dev.storj.io/trace/4195524620a3e2c2
FAIL (2025-12-08 15:42:09 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment
Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments...
Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces.
Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments...
Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded
Duration: 8.707531338s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [.............X............X.X.XX.XX.XX..X.....XX...X.X.X.X..] duration:
6.093414624s trace:
https://jaeger.dev.storj.io/trace/7b0090eb8f29bbc6
FAIL (2025-12-08 15:42:02 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.093480819s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
7.273860489s trace:
https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182
FAIL (2025-12-08 15:41:56 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 7.273890826s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
33 out of 60 is failed (threshold=6) [XXXXXXXXX.X.XXXXX.XXX..X....X...X...X.X.X.....XX.X.X.XXXX.X.] duration:
3.52306634s trace:
https://jaeger.dev.storj.io/trace/52dbb0834262bbe5
FAIL (2025-12-08 15:41:54 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.523098271s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.............................X.........................X..X.] duration:
5.768034686s trace:
https://jaeger.dev.storj.io/trace/43c87fd861da304
FAIL (2025-12-08 15:41:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.768055805s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..............................X.........................X..X] duration:
1m0.000739349s trace:
https://jaeger.dev.storj.io/trace/43c87fd861da304
FAIL (2025-12-08 15:41:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.00076131s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
34 out of 60 is failed (threshold=6) [XXXXXXXXXX.X.XXXXX.XXX..X....X...X...X.X.X.....XX.X.X.XXXX.X] duration:
5.001136764s trace:
https://jaeger.dev.storj.io/trace/52dbb0834262bbe5
FAIL (2025-12-08 15:41:39 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001160568s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [..............X............X.X.XX.XX.XX..X.....XX...X.X.X.X.] duration:
24.306964575s trace:
https://jaeger.dev.storj.io/trace/7b0090eb8f29bbc6
FAIL (2025-12-08 15:41:38 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 24.306997769s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
7.569667403s trace:
https://jaeger.dev.storj.io/trace/4195524620a3e2c2
FAIL (2025-12-08 15:41:30 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment
Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments...
Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces.
Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments...
Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded
Duration: 7.569694133s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.................................X.........X...X............] duration:
8.613652277s trace:
https://jaeger.dev.storj.io/trace/631c2b6153f16a63
FAIL (2025-12-08 15:41:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA) context deadline exceeded; piecestore: (Node ID: 13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, Piece ID: 5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A) context deadline exceeded; piecestore: (Node ID: 12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, Piece ID: DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA) context deadline exceeded
======LOGS======
Dec 8 15:31:47.733980955 (72h2m47.605064s) [upload=63079]: Starting upload
Dec 8 15:31:47.734113437 (72h2m47.605196482s) [upload=63079]: Done starting upload
Dec 8 15:31:47.734220060 (72h2m47.605303095s) [upload=63079]: Uploading segments...
Dec 8 15:31:47.734376698 (72h2m47.605459743s) [upload=63079]: Got next segment. Inline: false
Dec 8 15:31:47.734425131 (72h2m47.605508166s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:31:47.734431473 (72h2m47.605514518s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:31:48.240451445 (72h2m48.11153449s) [seg_pos={0 0}, upload=63079]: Begin upload segment...
Dec 8 15:31:48.240553519 (72h2m48.111636554s) [seg_pos={0 0}, upload=63079]: Done begin upload segment.
Dec 8 15:31:48.240584428 (72h2m48.111667473s) [noise=true , address=145.79.227.27:4691 , piece_id=TSGC3VYZETJX3AR7AFMC3WV5CIJAKQLVX6W2CA7PIZYYAOEZXYOA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.241909641 (72h2m48.112992686s) [noise=true , address=77.90.31.19:4841 , piece_id=QFLCO4T2GILZQ7QLAMQVMJMPNNPVXYX72XMFN2G3WLJ5MCSERAQQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.242622299 (72h2m48.113705334s) [noise=true , address=145.79.227.2:4113 , piece_id=VBI4ITMJCYEKRNQ2R5ZVL4PJYKYSMAGZDN7LJGXBT5DT5MKCS2EQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.243034072 (72h2m48.114117107s) [noise=true , address=145.79.227.31:4798 , piece_id=VIL36YB442SIWR6UHFDY4F62HHDJS2IB5T6YDOOAGMJIJEJNHZAQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.243641960 (72h2m48.114725005s) [noise=true , address=145.79.227.24:4581 , piece_id=JK2YXXKH2D33KLTYJ2JNW4I6HKQGJWR3BAUULSMUFKL4A2DHQZVA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244059034 (72h2m48.115142069s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244487580 (72h2m48.115570615s) [noise=true , address=145.79.227.20:4438 , piece_id=ZVQYBOT6NDTUNJFHWBBXBFALMQFFNZPHTVA2NIPJ2LGEOFLLOLEA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244706898 (72h2m48.115789944s) [noise=true , address=145.79.227.16:4910 , piece_id=JTWXUSJITQMTHBBRTRTSAYXNY3YN7H5WFAUTWAJEREMNMTDAB4RQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.245225194 (72h2m48.11630823s) [noise=true , address=145.79.227.41:4085 , piece_id=Z4NSLWRCQD4D6YHT7M27BGO3CLWK5WOFZ377G44SRUK4FJBA7GXA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.245653140 (72h2m48.116736185s) [noise=true , address=145.79.227.38:4977 , piece_id=Y626H42FV2KSMBMRZHC2S4R4DJLZA6I3XFEMMAOS62FTVRRZSS5A, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246124207 (72h2m48.117207242s) [noise=true , address=145.79.227.25:4618 , piece_id=ALUOULKKNPL56O2PATZO5ZQYSY6JLRAJ23Y4ZNGJBHCS4U3ELVWA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246531512 (72h2m48.117614547s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246921264 (72h2m48.118004299s) [noise=true , address=145.79.227.28:4726 , piece_id=GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.247439812 (72h2m48.118522847s) [noise=true , address=145.79.227.16:4911 , piece_id=5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.249604864 (72h2m48.120687909s) [noise=true , address=145.79.227.34:4905 , piece_id=EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.250349092 (72h2m48.121432137s) [noise=true , address=145.79.227.44:4195 , piece_id=DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:49.352808394 (72h2m49.223891439s) [noise=true , address=77.90.31.19:4841 , piece_id=QFLCO4T2GILZQ7QLAMQVMJMPNNPVXYX72XMFN2G3WLJ5MCSERAQQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 8 15:31:50.097967294 (72h2m49.969050339s) [upload=63079]: Next returned nil segment
Dec 8 15:31:50.097984627 (72h2m49.969067683s) [upload=63079]: Waiting for error group managing segments...
Dec 8 15:31:50.416319661 (72h2m50.287402706s) [noise=true , address=145.79.227.31:4798 , piece_id=VIL36YB442SIWR6UHFDY4F62HHDJS2IB5T6YDOOAGMJIJEJNHZAQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.427033041 (72h2m50.298116096s) [noise=true , address=145.79.227.16:4910 , piece_id=JTWXUSJITQMTHBBRTRTSAYXNY3YN7H5WFAUTWAJEREMNMTDAB4RQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.430562030 (72h2m50.301645115s) [noise=true , address=145.79.227.24:4581 , piece_id=JK2YXXKH2D33KLTYJ2JNW4I6HKQGJWR3BAUULSMUFKL4A2DHQZVA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.466400595 (72h2m50.33748364s) [noise=true , address=145.79.227.34:4905 , piece_id=EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.472873420 (72h2m50.343956455s) [noise=true , address=145.79.227.27:4691 , piece_id=TSGC3VYZETJX3AR7AFMC3WV5CIJAKQLVX6W2CA7PIZYYAOEZXYOA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.531272483 (72h2m50.402355528s) [noise=true , address=145.79.227.2:4113 , piece_id=VBI4ITMJCYEKRNQ2R5ZVL4PJYKYSMAGZDN7LJGXBT5DT5MKCS2EQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.729877357 (72h2m50.600960402s) [noise=true , address=145.79.227.16:4911 , piece_id=5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.779057345 (72h2m50.65014039s) [noise=true , address=145.79.227.25:4618 , piece_id=ALUOULKKNPL56O2PATZO5ZQYSY6JLRAJ23Y4ZNGJBHCS4U3ELVWA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.104680797 (72h2m50.975763832s) [noise=true , address=145.79.227.28:4726 , piece_id=GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.116683803 (72h2m50.987766848s) [noise=true , address=145.79.227.38:4977 , piece_id=Y626H42FV2KSMBMRZHC2S4R4DJLZA6I3XFEMMAOS62FTVRRZSS5A, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.132265363 (72h2m51.003348398s) [noise=true , address=145.79.227.20:4438 , piece_id=ZVQYBOT6NDTUNJFHWBBXBFALMQFFNZPHTVA2NIPJ2LGEOFLLOLEA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.146914038 (72h2m51.017997083s) [noise=true , address=145.79.227.44:4195 , piece_id=DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.452416276 (72h2m51.323499321s) [noise=true , address=145.79.227.41:4085 , piece_id=Z4NSLWRCQD4D6YHT7M27BGO3CLWK5WOFZ377G44SRUK4FJBA7GXA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.452447905 (72h2m51.32353095s) [seg_pos={0 0}, upload=63079]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:31:51.452802852 (72h2m51.323885897s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF
Dec 8 15:31:51.452816517 (72h2m51.323899562s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Long tail cancellation due to optimal threshold
Dec 8 15:31:51.452931045 (72h2m51.32401409s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: read tcp 188.166.225.170:41450->145.79.227.39:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:31:51.452966172 (72h2m51.324049217s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Long tail cancellation due to optimal threshold
Dec 8 15:31:51.452996309 (72h2m51.324079354s) [seg_pos={0 0}, upload=63079]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:31:51.453109756 (72h2m51.324192801s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:31:51.453116439 (72h2m51.324199484s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:31:52.077764883 (72h2m51.948847938s) [upload=63079]: Done uploading segments...
Dec 8 15:31:52.077787646 (72h2m51.948870691s) [upload=63079]: Upload finished. err: <nil>
Duration: 8.613733072s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
5.994578294s trace:
https://jaeger.dev.storj.io/trace/153706e03a29fd88
FAIL (2025-12-08 15:41:25 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 5.994668556s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [...............X............X.X.XX.XX.XX..X.....XX...X.X.X.X] duration:
30.000923237s trace:
https://jaeger.dev.storj.io/trace/7b0090eb8f29bbc6
FAIL (2025-12-08 15:41:12 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000948597s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
7.169307939s trace:
https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182
FAIL (2025-12-08 15:41:03 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 7.169403079s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..................................X.........X...X...........] duration:
8.803504486s trace:
https://jaeger.dev.storj.io/trace/631c2b6153f16a63
FAIL (2025-12-08 15:40:55 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA) context deadline exceeded; piecestore: (Node ID: 13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, Piece ID: 5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A) context deadline exceeded; piecestore: (Node ID: 12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, Piece ID: DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA) context deadline exceeded
======LOGS======
Dec 8 15:31:47.733980955 (72h2m47.605064s) [upload=63079]: Starting upload
Dec 8 15:31:47.734113437 (72h2m47.605196482s) [upload=63079]: Done starting upload
Dec 8 15:31:47.734220060 (72h2m47.605303095s) [upload=63079]: Uploading segments...
Dec 8 15:31:47.734376698 (72h2m47.605459743s) [upload=63079]: Got next segment. Inline: false
Dec 8 15:31:47.734425131 (72h2m47.605508166s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:31:47.734431473 (72h2m47.605514518s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:31:48.240451445 (72h2m48.11153449s) [seg_pos={0 0}, upload=63079]: Begin upload segment...
Dec 8 15:31:48.240553519 (72h2m48.111636554s) [seg_pos={0 0}, upload=63079]: Done begin upload segment.
Dec 8 15:31:48.240584428 (72h2m48.111667473s) [noise=true , address=145.79.227.27:4691 , piece_id=TSGC3VYZETJX3AR7AFMC3WV5CIJAKQLVX6W2CA7PIZYYAOEZXYOA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.241909641 (72h2m48.112992686s) [noise=true , address=77.90.31.19:4841 , piece_id=QFLCO4T2GILZQ7QLAMQVMJMPNNPVXYX72XMFN2G3WLJ5MCSERAQQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.242622299 (72h2m48.113705334s) [noise=true , address=145.79.227.2:4113 , piece_id=VBI4ITMJCYEKRNQ2R5ZVL4PJYKYSMAGZDN7LJGXBT5DT5MKCS2EQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.243034072 (72h2m48.114117107s) [noise=true , address=145.79.227.31:4798 , piece_id=VIL36YB442SIWR6UHFDY4F62HHDJS2IB5T6YDOOAGMJIJEJNHZAQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.243641960 (72h2m48.114725005s) [noise=true , address=145.79.227.24:4581 , piece_id=JK2YXXKH2D33KLTYJ2JNW4I6HKQGJWR3BAUULSMUFKL4A2DHQZVA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244059034 (72h2m48.115142069s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244487580 (72h2m48.115570615s) [noise=true , address=145.79.227.20:4438 , piece_id=ZVQYBOT6NDTUNJFHWBBXBFALMQFFNZPHTVA2NIPJ2LGEOFLLOLEA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244706898 (72h2m48.115789944s) [noise=true , address=145.79.227.16:4910 , piece_id=JTWXUSJITQMTHBBRTRTSAYXNY3YN7H5WFAUTWAJEREMNMTDAB4RQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.245225194 (72h2m48.11630823s) [noise=true , address=145.79.227.41:4085 , piece_id=Z4NSLWRCQD4D6YHT7M27BGO3CLWK5WOFZ377G44SRUK4FJBA7GXA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.245653140 (72h2m48.116736185s) [noise=true , address=145.79.227.38:4977 , piece_id=Y626H42FV2KSMBMRZHC2S4R4DJLZA6I3XFEMMAOS62FTVRRZSS5A, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246124207 (72h2m48.117207242s) [noise=true , address=145.79.227.25:4618 , piece_id=ALUOULKKNPL56O2PATZO5ZQYSY6JLRAJ23Y4ZNGJBHCS4U3ELVWA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246531512 (72h2m48.117614547s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246921264 (72h2m48.118004299s) [noise=true , address=145.79.227.28:4726 , piece_id=GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.247439812 (72h2m48.118522847s) [noise=true , address=145.79.227.16:4911 , piece_id=5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.249604864 (72h2m48.120687909s) [noise=true , address=145.79.227.34:4905 , piece_id=EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.250349092 (72h2m48.121432137s) [noise=true , address=145.79.227.44:4195 , piece_id=DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:49.352808394 (72h2m49.223891439s) [noise=true , address=77.90.31.19:4841 , piece_id=QFLCO4T2GILZQ7QLAMQVMJMPNNPVXYX72XMFN2G3WLJ5MCSERAQQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 8 15:31:50.097967294 (72h2m49.969050339s) [upload=63079]: Next returned nil segment
Dec 8 15:31:50.097984627 (72h2m49.969067683s) [upload=63079]: Waiting for error group managing segments...
Dec 8 15:31:50.416319661 (72h2m50.287402706s) [noise=true , address=145.79.227.31:4798 , piece_id=VIL36YB442SIWR6UHFDY4F62HHDJS2IB5T6YDOOAGMJIJEJNHZAQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.427033041 (72h2m50.298116096s) [noise=true , address=145.79.227.16:4910 , piece_id=JTWXUSJITQMTHBBRTRTSAYXNY3YN7H5WFAUTWAJEREMNMTDAB4RQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.430562030 (72h2m50.301645115s) [noise=true , address=145.79.227.24:4581 , piece_id=JK2YXXKH2D33KLTYJ2JNW4I6HKQGJWR3BAUULSMUFKL4A2DHQZVA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.466400595 (72h2m50.33748364s) [noise=true , address=145.79.227.34:4905 , piece_id=EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.472873420 (72h2m50.343956455s) [noise=true , address=145.79.227.27:4691 , piece_id=TSGC3VYZETJX3AR7AFMC3WV5CIJAKQLVX6W2CA7PIZYYAOEZXYOA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.531272483 (72h2m50.402355528s) [noise=true , address=145.79.227.2:4113 , piece_id=VBI4ITMJCYEKRNQ2R5ZVL4PJYKYSMAGZDN7LJGXBT5DT5MKCS2EQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.729877357 (72h2m50.600960402s) [noise=true , address=145.79.227.16:4911 , piece_id=5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.779057345 (72h2m50.65014039s) [noise=true , address=145.79.227.25:4618 , piece_id=ALUOULKKNPL56O2PATZO5ZQYSY6JLRAJ23Y4ZNGJBHCS4U3ELVWA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.104680797 (72h2m50.975763832s) [noise=true , address=145.79.227.28:4726 , piece_id=GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.116683803 (72h2m50.987766848s) [noise=true , address=145.79.227.38:4977 , piece_id=Y626H42FV2KSMBMRZHC2S4R4DJLZA6I3XFEMMAOS62FTVRRZSS5A, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.132265363 (72h2m51.003348398s) [noise=true , address=145.79.227.20:4438 , piece_id=ZVQYBOT6NDTUNJFHWBBXBFALMQFFNZPHTVA2NIPJ2LGEOFLLOLEA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.146914038 (72h2m51.017997083s) [noise=true , address=145.79.227.44:4195 , piece_id=DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.452416276 (72h2m51.323499321s) [noise=true , address=145.79.227.41:4085 , piece_id=Z4NSLWRCQD4D6YHT7M27BGO3CLWK5WOFZ377G44SRUK4FJBA7GXA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.452447905 (72h2m51.32353095s) [seg_pos={0 0}, upload=63079]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:31:51.452802852 (72h2m51.323885897s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF
Dec 8 15:31:51.452816517 (72h2m51.323899562s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Long tail cancellation due to optimal threshold
Dec 8 15:31:51.452931045 (72h2m51.32401409s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: read tcp 188.166.225.170:41450->145.79.227.39:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:31:51.452966172 (72h2m51.324049217s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Long tail cancellation due to optimal threshold
Dec 8 15:31:51.452996309 (72h2m51.324079354s) [seg_pos={0 0}, upload=63079]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:31:51.453109756 (72h2m51.324192801s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:31:51.453116439 (72h2m51.324199484s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:31:52.077764883 (72h2m51.948847938s) [upload=63079]: Done uploading segments...
Dec 8 15:31:52.077787646 (72h2m51.948870691s) [upload=63079]: Upload finished. err: <nil>
Duration: 8.803534674s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
34 out of 60 is failed (threshold=6) [XXXXXXXXXXX.X.XXXXX.XXX..X....X...X...X.X.X.....XX.X.X.XXXX.] duration:
3.350260903s trace:
https://jaeger.dev.storj.io/trace/4813cfd3eeabd4ff
FAIL (2025-12-08 15:40:52 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.350292712s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
4.759130115s trace:
https://jaeger.dev.storj.io/trace/4195524620a3e2c2
FAIL (2025-12-08 15:40:28 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment
Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments...
Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces.
Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments...
Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded
Duration: 4.75915386s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
7.732761838s trace:
https://jaeger.dev.storj.io/trace/54ba7b1e9d79a182
FAIL (2025-12-08 15:40:27 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 7.732792127s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
5.72656179s trace:
https://jaeger.dev.storj.io/trace/153706e03a29fd88
FAIL (2025-12-08 15:40:23 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 5.7265882s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [...............................X.........................X..] duration:
5.800157714s trace:
https://jaeger.dev.storj.io/trace/3b9a17c2d9c87842
FAIL (2025-12-08 15:40:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.800187932s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
35 out of 60 is failed (threshold=6) [XXXXXXXXXXXX.X.XXXXX.XXX..X....X...X...X.X.X.....XX.X.X.XXXX] duration:
5.001561572s trace:
https://jaeger.dev.storj.io/trace/4813cfd3eeabd4ff
FAIL (2025-12-08 15:40:20 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001576512s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [................................X.........................X.] duration:
19.519192991s trace:
https://jaeger.dev.storj.io/trace/3b9a17c2d9c87842
FAIL (2025-12-08 15:40:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 19.519589175s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...................................X.........X...X..........] duration:
9.838801074s trace:
https://jaeger.dev.storj.io/trace/631c2b6153f16a63
FAIL (2025-12-08 15:39:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4, Piece ID: GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA) context deadline exceeded; piecestore: (Node ID: 13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, Piece ID: 5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ) context deadline exceeded; piecestore: (Node ID: 12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9, Piece ID: EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A) context deadline exceeded; piecestore: (Node ID: 12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, Piece ID: DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA) context deadline exceeded
======LOGS======
Dec 8 15:31:47.733980955 (72h2m47.605064s) [upload=63079]: Starting upload
Dec 8 15:31:47.734113437 (72h2m47.605196482s) [upload=63079]: Done starting upload
Dec 8 15:31:47.734220060 (72h2m47.605303095s) [upload=63079]: Uploading segments...
Dec 8 15:31:47.734376698 (72h2m47.605459743s) [upload=63079]: Got next segment. Inline: false
Dec 8 15:31:47.734425131 (72h2m47.605508166s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 15:31:47.734431473 (72h2m47.605514518s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 15:31:48.240451445 (72h2m48.11153449s) [seg_pos={0 0}, upload=63079]: Begin upload segment...
Dec 8 15:31:48.240553519 (72h2m48.111636554s) [seg_pos={0 0}, upload=63079]: Done begin upload segment.
Dec 8 15:31:48.240584428 (72h2m48.111667473s) [noise=true , address=145.79.227.27:4691 , piece_id=TSGC3VYZETJX3AR7AFMC3WV5CIJAKQLVX6W2CA7PIZYYAOEZXYOA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.241909641 (72h2m48.112992686s) [noise=true , address=77.90.31.19:4841 , piece_id=QFLCO4T2GILZQ7QLAMQVMJMPNNPVXYX72XMFN2G3WLJ5MCSERAQQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.242622299 (72h2m48.113705334s) [noise=true , address=145.79.227.2:4113 , piece_id=VBI4ITMJCYEKRNQ2R5ZVL4PJYKYSMAGZDN7LJGXBT5DT5MKCS2EQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.243034072 (72h2m48.114117107s) [noise=true , address=145.79.227.31:4798 , piece_id=VIL36YB442SIWR6UHFDY4F62HHDJS2IB5T6YDOOAGMJIJEJNHZAQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.243641960 (72h2m48.114725005s) [noise=true , address=145.79.227.24:4581 , piece_id=JK2YXXKH2D33KLTYJ2JNW4I6HKQGJWR3BAUULSMUFKL4A2DHQZVA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244059034 (72h2m48.115142069s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244487580 (72h2m48.115570615s) [noise=true , address=145.79.227.20:4438 , piece_id=ZVQYBOT6NDTUNJFHWBBXBFALMQFFNZPHTVA2NIPJ2LGEOFLLOLEA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.244706898 (72h2m48.115789944s) [noise=true , address=145.79.227.16:4910 , piece_id=JTWXUSJITQMTHBBRTRTSAYXNY3YN7H5WFAUTWAJEREMNMTDAB4RQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.245225194 (72h2m48.11630823s) [noise=true , address=145.79.227.41:4085 , piece_id=Z4NSLWRCQD4D6YHT7M27BGO3CLWK5WOFZ377G44SRUK4FJBA7GXA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.245653140 (72h2m48.116736185s) [noise=true , address=145.79.227.38:4977 , piece_id=Y626H42FV2KSMBMRZHC2S4R4DJLZA6I3XFEMMAOS62FTVRRZSS5A, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246124207 (72h2m48.117207242s) [noise=true , address=145.79.227.25:4618 , piece_id=ALUOULKKNPL56O2PATZO5ZQYSY6JLRAJ23Y4ZNGJBHCS4U3ELVWA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246531512 (72h2m48.117614547s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.246921264 (72h2m48.118004299s) [noise=true , address=145.79.227.28:4726 , piece_id=GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.247439812 (72h2m48.118522847s) [noise=true , address=145.79.227.16:4911 , piece_id=5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.249604864 (72h2m48.120687909s) [noise=true , address=145.79.227.34:4905 , piece_id=EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:48.250349092 (72h2m48.121432137s) [noise=true , address=145.79.227.44:4195 , piece_id=DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA, seg_pos={0 0}, upload=63079]: Uploading piece...
Dec 8 15:31:49.352808394 (72h2m49.223891439s) [noise=true , address=77.90.31.19:4841 , piece_id=QFLCO4T2GILZQ7QLAMQVMJMPNNPVXYX72XMFN2G3WLJ5MCSERAQQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 8 15:31:50.097967294 (72h2m49.969050339s) [upload=63079]: Next returned nil segment
Dec 8 15:31:50.097984627 (72h2m49.969067683s) [upload=63079]: Waiting for error group managing segments...
Dec 8 15:31:50.416319661 (72h2m50.287402706s) [noise=true , address=145.79.227.31:4798 , piece_id=VIL36YB442SIWR6UHFDY4F62HHDJS2IB5T6YDOOAGMJIJEJNHZAQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.427033041 (72h2m50.298116096s) [noise=true , address=145.79.227.16:4910 , piece_id=JTWXUSJITQMTHBBRTRTSAYXNY3YN7H5WFAUTWAJEREMNMTDAB4RQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.430562030 (72h2m50.301645115s) [noise=true , address=145.79.227.24:4581 , piece_id=JK2YXXKH2D33KLTYJ2JNW4I6HKQGJWR3BAUULSMUFKL4A2DHQZVA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.466400595 (72h2m50.33748364s) [noise=true , address=145.79.227.34:4905 , piece_id=EIDUJOUHNB3SV3DCUFAHZTZCR2N4EXLHFJHTRG3FTXXITRVIWT5A, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.472873420 (72h2m50.343956455s) [noise=true , address=145.79.227.27:4691 , piece_id=TSGC3VYZETJX3AR7AFMC3WV5CIJAKQLVX6W2CA7PIZYYAOEZXYOA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.531272483 (72h2m50.402355528s) [noise=true , address=145.79.227.2:4113 , piece_id=VBI4ITMJCYEKRNQ2R5ZVL4PJYKYSMAGZDN7LJGXBT5DT5MKCS2EQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.729877357 (72h2m50.600960402s) [noise=true , address=145.79.227.16:4911 , piece_id=5CQVH54BNZCST7AMPW5YKFGIEKJY5D2LJJWBWS4CASNJ7ROUH7VQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:50.779057345 (72h2m50.65014039s) [noise=true , address=145.79.227.25:4618 , piece_id=ALUOULKKNPL56O2PATZO5ZQYSY6JLRAJ23Y4ZNGJBHCS4U3ELVWA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.104680797 (72h2m50.975763832s) [noise=true , address=145.79.227.28:4726 , piece_id=GHYOKORZKJC22XWUKLHHNLLPBGJUIOHFPKDEYCSD62VL5TQLZJDA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.116683803 (72h2m50.987766848s) [noise=true , address=145.79.227.38:4977 , piece_id=Y626H42FV2KSMBMRZHC2S4R4DJLZA6I3XFEMMAOS62FTVRRZSS5A, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.132265363 (72h2m51.003348398s) [noise=true , address=145.79.227.20:4438 , piece_id=ZVQYBOT6NDTUNJFHWBBXBFALMQFFNZPHTVA2NIPJ2LGEOFLLOLEA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.146914038 (72h2m51.017997083s) [noise=true , address=145.79.227.44:4195 , piece_id=DPHMZWVSUDEAEKRMVEHXS47BTEUPRVW7EYC2YNLRGF5FNSSMTIEA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.452416276 (72h2m51.323499321s) [noise=true , address=145.79.227.41:4085 , piece_id=Z4NSLWRCQD4D6YHT7M27BGO3CLWK5WOFZ377G44SRUK4FJBA7GXA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: <nil>
Dec 8 15:31:51.452447905 (72h2m51.32353095s) [seg_pos={0 0}, upload=63079]: Segment reached optimal threshold of 13 pieces.
Dec 8 15:31:51.452802852 (72h2m51.323885897s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF
Dec 8 15:31:51.452816517 (72h2m51.323899562s) [noise=true , address=145.79.227.29:4941 , piece_id=P3FZJDKNLKQHSMF4QL3XJGZFSX2GBYGVYZZ7VBAVJIRHJCYMPKDA, seg_pos={0 0}, upload=63079]: Long tail cancellation due to optimal threshold
Dec 8 15:31:51.452931045 (72h2m51.32401409s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121scLvKEW9zw2aEz22zpremcjiop67eQ8v2uqpmaD5aZzCAP7b): protocol: read tcp 188.166.225.170:41450->145.79.227.39:4013: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 8 15:31:51.452966172 (72h2m51.324049217s) [noise=true , address=145.79.227.39:4013 , piece_id=JC7DFE7N3NJVHTPXULEI6WLUYZRPMCCZMFF62JONDTNNAEAC4WCQ, seg_pos={0 0}, upload=63079]: Long tail cancellation due to optimal threshold
Dec 8 15:31:51.452996309 (72h2m51.324079354s) [seg_pos={0 0}, upload=63079]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 15:31:51.453109756 (72h2m51.324192801s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:31:51.453116439 (72h2m51.324199484s) [upload=63079]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:31:52.077764883 (72h2m51.948847938s) [upload=63079]: Done uploading segments...
Dec 8 15:31:52.077787646 (72h2m51.948870691s) [upload=63079]: Upload finished. err: <nil>
Duration: 9.838828005s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [................X............X.X.XX.XX.XX..X.....XX...X.X.X.] duration:
20.500344522s trace:
https://jaeger.dev.storj.io/trace/3cf04ebc2c693eb9
FAIL (2025-12-08 15:39:55 UTC)
orbiter warning: context deadline exceeded
Duration: 20.500372054s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
4.338074838s trace:
https://jaeger.dev.storj.io/trace/4195524620a3e2c2
FAIL (2025-12-08 15:39:49 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
, address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771167108 (72h9m11.642250153s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771462930 (72h9m11.642545976s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.771589993 (72h9m11.642673038s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772028248 (72h9m11.643111293s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772041683 (72h9m11.643124728s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.772603803 (72h9m11.643686848s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773136077 (72h9m11.644219122s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.773659244 (72h9m11.644742289s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774276288 (72h9m11.645359333s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:11.774811648 (72h9m11.645894693s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Uploading piece...
Dec 8 15:38:12.600277390 (72h9m12.471360435s) [upload=63171]: Next returned nil segment
Dec 8 15:38:12.600300113 (72h9m12.471383158s) [upload=63171]: Waiting for error group managing segments...
Dec 8 15:38:12.984985581 (72h9m12.856068647s) [noise=true , address=185.167.204.205:4028 , piece_id=DCGYFXSKGK6GFLLFU7UQONAX643EA6CDTYWEZ42YQPLEERINL3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.985372348 (72h9m12.856455383s) [noise=true , address=185.167.204.205:4008 , piece_id=JWZAQAYIUNBFRDVQHADARS5NTEWET7UI2GR73ZMIACA5VFUS3D6Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.993832749 (72h9m12.864915784s) [noise=true , address=185.167.204.205:4015 , piece_id=GK2HV5AGF2Q6RYHDZBZQ3SY537WQXB7E7UF4SWF5X3LFM4PMCZPQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994210087 (72h9m12.865293122s) [noise=true , address=185.167.204.206:4053 , piece_id=DGSLZPKB474OADK43GBZO2FGAR2F62PBL364EWVNPCZE3PWZGQPA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994561226 (72h9m12.865644271s) [noise=true , address=185.167.204.205:4036 , piece_id=X2TWRAEKE6FEH3ITYY74A76ZZQ2KCB63ETZNW5TAHQ3MLEM34PZQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.994944356 (72h9m12.866027401s) [noise=true , address=185.167.204.205:4023 , piece_id=5RNGVQ3LUTPKX4RAVK7Y4WRQC2Y6GT5QWV2G3MTY2LW2MU6IMKLQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995089352 (72h9m12.866172397s) [noise=true , address=185.167.204.205:4007 , piece_id=WEZIGATHDFL7G3S2VG7BEFGGT7JJIXBDWHGIPHBTPSKDJQUTROIQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:12.995157291 (72h9m12.866240336s) [noise=true , address=185.167.204.205:4033 , piece_id=4V3DXW42ZCPJFF3PPJEBMBYFW354WEZYLVFUSPGEZ4E4HZ7MFI4Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000801098 (72h9m12.871884163s) [noise=true , address=185.167.204.206:4062 , piece_id=KYFUGH5EAVFGEJ2J7VDRDWBDP2R2MX4JSCIU7RP3VJOK4LCM6D2Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.000862164 (72h9m12.871945209s) [noise=true , address=185.167.204.206:4051 , piece_id=5JMUWS6LLPRENPBAZ74MDELLIZMWTMI3GOZFUWLCOCUNLDTPETWQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.002117705 (72h9m12.87320074s) [noise=true , address=185.167.204.206:4064 , piece_id=LUITKU5LPQ67MFA4VWE6CDV46QOU7RXOFCGKKAYPCLFGWFSLYLTA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003015134 (72h9m12.874098179s) [noise=true , address=185.167.204.205:4034 , piece_id=OJPEC5SBHQDKJW3LDBPHMEOPDG6NPNNMO3W54C7XIXTNUYBTMUOA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.003432589 (72h9m12.874515624s) [noise=true , address=185.167.204.206:4037 , piece_id=OPF6NTGFZ7FSN7LIEKHOGIJRYUWX27RTI3KKGCKPCOQLK4HC62LQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.004078648 (72h9m12.875161694s) [noise=true , address=185.167.204.206:4069 , piece_id=GH2ZEUIEIUZ6OLM6XH5ZJVXKAMHUKQ6L3GZ7NJLUZFY623YH5GDQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.009788571 (72h9m12.880871616s) [noise=true , address=185.167.204.206:4067 , piece_id=3YOG2SUYBHBE6MMIPMQWMBDZTLL7HPSBPEKFHMGOICBTAQU5HA4A, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.012083561 (72h9m12.883166596s) [noise=true , address=185.167.204.205:4030 , piece_id=EFAUOQCDA25VE55ZHJOSVXTOQD2VTQCW55MEH5DBEAY7N364GMXA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.584995241 (72h9m13.456078286s) [noise=true , address=81.171.14.70:4046 , piece_id=77RPCXTIQW7MO6UKAVJUD4Y7XR674IDZE2TL2SB77LJUHRE3ACDA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.651263008 (72h9m13.522346063s) [noise=true , address=81.171.14.70:4026 , piece_id=4BLGOA4GSQVOKZ6UBUIXANALBDHDHDXPLDA3TDQ44ACTIC2IHSEA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:13.696266204 (72h9m13.567349249s) [noise=true , address=81.171.14.70:4035 , piece_id=VYAAT75X4R4J22MLY7RK4VHOEFMYZTLWUWTWVAMOOIIDSEXTQV7Q, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.123999844 (72h9m13.995082889s) [noise=true , address=81.171.14.70:4025 , piece_id=XML4X66PHAT33VTUTEV22HSTIHMG4WFZU6YSDPCZZZ4SEBL3NWUA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.147808461 (72h9m14.018891506s) [noise=true , address=81.171.14.70:4032 , piece_id=XAMDRY73RIJV4LC2W2FRMH6L4WL3VAFUYKB3QD4TXOQWWZUYGPOQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.255119813 (72h9m14.126202858s) [noise=true , address=89.149.201.147:4005 , piece_id=MBR3CYKQ3YSPVXIVSL6ABGZNG7FB5HGP45OA3YE2IRKPJMOSM3KQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.271452473 (72h9m14.142535518s) [noise=true , address=89.149.201.147:4013 , piece_id=KIPMWZVYZADDS2LAOY5ASMIWP32HIRTNGYQ65SNLYR2EQDQCN3HQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.775652542 (72h9m14.646735587s) [noise=true , address=89.149.201.147:4016 , piece_id=5N5HKBPMDQUEQIZZVFFRQGE4LQAGSJ4H6INPDXHYQMDK5YTHUVKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.785904314 (72h9m14.656987359s) [noise=true , address=89.149.201.147:4017 , piece_id=MBD47F6TJHZ76PAGU7ZIFRQ4M5HUDT5AQULHE6RPQPBUBAVKSTKQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827349233 (72h9m14.698432278s) [noise=true , address=89.149.201.147:4022 , piece_id=CWWNRXZNWQOUTZEA4WPHGA6BYEPO35D2YDU5VSTHFZ7GRC67Y7VQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: <nil>
Dec 8 15:38:14.827365925 (72h9m14.69844897s) [seg_pos={0 0}, upload=63171]: Segment reached optimal threshold of 26 pieces.
Dec 8 15:38:14.827539125 (72h9m14.69862217s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hJ5jT6S2d3DJe9DoVDjHMkpwSTpeyTMyNLaK8Pk8SWcUVRKnv): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827545567 (72h9m14.698628612s) [noise=true , address=89.149.201.147:4012 , piece_id=OVDDLTF3BGCEBUGICJRMVFEL6577S5NHNUHVARGOT6FF4GLSM7RA, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827903338 (72h9m14.698986373s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Ku7o12bSmJ1YR3dGetGFcWqs1wJEh53Zkg3H7ELJrt4w1hkua): protocol: expected piece hash; context canceled; EOF
Dec 8 15:38:14.827910673 (72h9m14.698993708s) [noise=true , address=89.149.201.147:4018 , piece_id=BFJH5ELBTZD767FJNFT2ZBUS3GI62IZIRSKYARLVBCLWZONSURMQ, seg_pos={0 0}, upload=63171]: Long tail cancellation due to optimal threshold
Dec 8 15:38:14.827928065 (72h9m14.69901111s) [seg_pos={0 0}, upload=63171]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 8 15:38:14.828037083 (72h9m14.699120128s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 15:38:14.828053224 (72h9m14.699136269s) [upload=63171]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 15:39:18.272006268 (72h10m18.143089313s) [upload=63171]: Done uploading segments...
Dec 8 15:39:18.272087593 (72h10m18.143170628s) [upload=63171]: Upload finished. err: metaclient: context deadline exceeded
Duration: 4.338105436s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
12.383416864s trace:
https://jaeger.dev.storj.io/trace/153706e03a29fd88
FAIL (2025-12-08 15:39:40 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 12.383439758s
edge
gateway
us-select-1
|